Introduction
For every test run, we had a requirement to download application log files from Azure blob storage, parse and generate performance metrics (response times).
Problem statement/Challenge
Our performance tests run for longer duration like (3hours/24hours/5days) and write thousands/lakhs of logs files to blob storage.
We have to filter and download logs based on different criteria’s like
- Download Logs only after particular Timestamp in UTC.
- Download all logs.
- Download only those logs, in which file name contains given string.
- Download only those log file with specific extension.
Solution
I had written below C# code snippet that will automatically download logs based on different criteria. This code snippet can be further customized and reused based on our needs.
Code snippet along with explanation in comments
App.Config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key = "StorageAccountConnectionString"
value = "DefaultEndpointsProtocol=https;AccountName=testaccount;AccountKey=testaccountkey+sy5AthQhOBRw==" />
<add key = "ContainerName" value = "testcontainername" />
<add key = "TestExecutionStartTimeinIST" value = "9/18/2015 10:30" />
<!– To download all files, use * in value
Example: add key = "NameContains" value = "*”
To download file that contains a give string in name, use string name
Example: add key = "NameContains" value = "Application”
To download file with extention, use extention
Example: add key = "NameContains" value = ".txt”
– >
<add key = "NameContains" value = "Application" />
<add key = "FileExtention" value = ".txt" />
</appSettings>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
</configuration>
Program.cs
// Read configuration from app.config
string strgConnString = ConfigurationManager.AppSettings["StorageAccountConnectionString"];
string containerName = ConfigurationManager.AppSettings["ContainerName"];
string fileNameMatch = ConfigurationManager.AppSettings["NameContains"];
string fileExtension = ConfigurationManager.AppSettings["FileExtention"];
string dateInputinUTC = ConfigurationManager.AppSettings["TestExecutionStartTimeinIST"];
// Create a Storage Account
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(strgConnString);
// Create the blob client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Get reference to a container
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
string filename = string.Empty;
// Convert time to UTC
DateTime configtime = DateTime.Parse(dateInputinUTC).ToUniversalTime();
Console.WriteLine("*******************Downloading Files from blob after :{0}", configtime);
// Traverse and download each item matching filter criteria
foreach (IListBlobItem item in container.ListBlobs(null, false))
{
if (item.GetType() == typeof(CloudBlockBlob))
{
CloudBlockBlob blob = (CloudBlockBlob)item;
// Condition to download files based on last modified
if (blob.Properties.LastModified.Value.Date >= configtime.Date)
{
if (blob.Properties.LastModified.Value.TimeOfDay >= configtime.TimeOfDay)
{
// Download all files
if (fileNameMatch == "*")
{
Console.WriteLine(blob.Uri.AbsolutePath);
string path = @"DownloadFolder\" + filename;
using (var fileStream = System.IO.File.Create(path))
{
blob.DownloadToStream(fileStream);
}
}
// Download all files that contains given string
else if (blob.Name.Contains(fileNameMatch) == true)
{
Console.WriteLine(blob.Uri.AbsolutePath);
filename = blob.Uri.AbsolutePath.Split('/')[2];
string path = @"DownloadFolder\" + filename;
using (var fileStream = System.IO.File.Create(path))
{
blob.DownloadToStream(fileStream);
}
}
// Download all files with extension
else if (blob.Name.EndsWith(fileExtension))
{
Console.WriteLine(blob.Uri.AbsolutePath);
filename = blob.Uri.AbsolutePath.Split('/')[2];
string path = @"DownloadFolder\" + filename;
using (var fileStream = System.IO.File.Create(path))
{
blob.DownloadToStream(fileStream);
}
}
}
}
}
}
Hope this will help you. Suggestion and Feedback is welcome.
Happy Coding!
References
https://azure.microsoft.com/en-in/documentation/articles/storage-dotnet-how-to-use-blobs/
