This article is a mirror article of machine translation, please click here to jump to the original article.

Architect_Programmer_Code Agriculture Network»Architect › Programming › .Net/C # › .NET/C# High performance Merge multiple files into a single file ...
View: 3255|Reply: 2

[Source] .NET/C# High performance merges multiple files into a single file

[Copy link]
Posted on 2024-2-14 12:04:26 | | | |
Requirements: When using large files to upload in chunks, the backend needs to merge different blocks into a single file in order, or when cutting the archive file, merge the cut files into the source file.

Linux uses split file cutting splits
https://www.itsvse.com/thread-10716-1-1.html

First, prepare two 100M files for testing, with the following command:

Windows creates a file of the specified size
https://www.itsvse.com/thread-10732-1-1.html

The code is as follows:


Two different merge scenarios were tested. As shown below:



Start testing with VS 2022 debug -> Profiler -> tick ".NET Object Allocation Trace":



merge1 total size is: 16,778,034, merge2 total size is: 164,270,The total allocation shows that merge1 is 102 times larger than merge2, and the larger it is, the worse the performance。

Use PerfView to view CPU Stacks and GC Heap Net Mem Stacks analysis separately, as shown in the following figure:



Remark:Inc% - Inclusive cost is expressed as a percentage of the total cost for all samples(100% for ROOT nodes). It can be sorted by it.





Previous:Windows creates a file of the specified size
Next:Introduction and difference between JIT and AOT compilation in .NET

Related Posts
 Landlord| Posted on 2024-2-14 12:09:30 |
Test with VS 2022 Debug -> Performance Profiler -> tick "Memory Usage":

  • merge1 takes up 808MB when executed
  • merge2 takes up 12MB when executed




If the merge2 method is executed alone, the program starts up 6MB and always takes up 6MB during function execution and after completion.
 Landlord| Posted on 2025-1-1 10:20:06 |
Optimize the writing style

Disclaimer:
All software, programming materials or articles published by Code Farmer Network are only for learning and research purposes; The above content shall not be used for commercial or illegal purposes, otherwise, users shall bear all consequences. The information on this site comes from the Internet, and copyright disputes have nothing to do with this site. You must completely delete the above content from your computer within 24 hours of downloading. If you like the program, please support genuine software, purchase registration, and get better genuine services. If there is any infringement, please contact us by email.
Mail To:help@itsvse.com