HTML5's FileReader API allows the client browser to read the user's local files, so that the uploaded file is no longer read by the server, which greatly reduces the burden on the server and saves the time required to upload the file. However, in practice, I found that I can easily handle a 300k log file with FileReader.readAsText(), but when the log file is as large as 1G or even 2G, the browser will crash. This is because readAsText() will load the target file into memory all at once, causing the memory to exceed the limit. So if the web application often needs to process large files, we should use FileReader.readAsArrayBuffer() to read the files piece by piece.
Test scenario
Our scenario is simple, which is to use JavaScript to get the time range of an IIS log
Sample IIS logs:
#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2016-08-18 06:53:55
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2016-08-18 06:53:55 ::1 GET / - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0; +WOW64; +Trident/7.0; +rv:11.0)+like+Gecko - 200 0 0 476
2016-08-18 06:53:55 ::1 GET /iisstart.png - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0; +WOW64; +Trident/7.0; +rv:11.0)+like+Gecko http://localhost/ 200 0 0 3
2016-08-18 08:45:34 10.172.19.198 GET /test/pac/wpad.dat - 80 - 10.157.21.235 Mozilla/5.0+(Windows+NT+6.1; +Win64; +x64; +Trident/7.0; +rv:11.0)+like+Gecko - 404 3 50 265
2016-08-18 08:46:44 10.172.19.198 GET /test/pac/wpad.dat - 80 - 10.157.21.235 Mozilla/5.0+(Windows+NT+6.1; +Win64; +x64; +Trident/7.0; +rv:11.0)+like+Gecko - 200 0 0 6
Start time: 2016-08-18 06:53:55
End time: 2016-08-18 08:46:44
Use readAsText() implementation
Using readAsText() is relatively simple, after getting the string of the entire file, get the first 19 characters of each line from the beginning, determine whether the date format is met, if it is satisfied, then these 19 characters are the start time, and the same goes through each line from the tail to get the end time, the code is as follows:
The running results of the sample IIS log (size: 1k) are as we expected.
But once we chooseA larger IIS log (size: 2G) and the browser crashes。 The reason is that readAsText() will load the entire file into memory first, so if the file is too large, there will not be enough memory and the browser process will crash.
Use readAsArrayBuffer() implementation
Since the File object in JavaScript inherits from Blob, we can use the Blob.slice() method to cut the file into small pieces, the general idea is:
First, take the first 10k contents of the file and convert it to text
Take the first 19 characters of each line from the beginning to determine if the date format is met, and if so, these 19 characters are the start time
Then take the 10k content at the end of the file and convert it into text
Similarly, traverse each line from the tail content to get the end time
The code is as follows:
Using readAsArrayBuffer(), we were able to get the results we wanted in a very short time, even with more than 2G IIS logs.
|
Posted on 6/2/2019 8:35:17 PM
|
|
|
|
Landlord|
Posted on 6/2/2019 8:39:24 PM
|
Posted on 9/17/2021 9:11:00 AM
|
