It was not easy to make this dictionary, first it took me a day to package and upload all kinds of dictionaries that I had stored on the server; and packaged and uploaded the dictionaries produced by netizens in this forum for another night;
The dictionary processing server is borrowed from a friend, and the friend's new webserver needs to be a copying machine, so I lied that I made a dictionary to repeat and detect the CPU, and my friend gladly lent it to me for three days; Configuration:
Four Xeon E3 3.10GHZ /32G DDR3 RAM / 250G SSD hard drives; I reinstalled Debian 6.0 64-bit;
Not counting uploads and downloads, this dictionary goes through:
1. Merge, a total of 101.6G large files (but later proved to be terrible of repetition);
2. Remove duplicate content, this is the first time-consuming work, it took more than 30 hours to process the 101.6G file, of which the previous leak door 21 stations I found to make a total of about 100G when the content was uncertain, and the weak password was more than 1G;
3. Go to characters other than Chinese and ASCII encoding, here I took a few detours but after testing it several times on the machine before daring to write it in batch, this time is the second time-consuming work. In order to speed up this step, I divided the dictionary processed in the second step into ten parts, and ran batch processing at the same time, which took about 25 hours;
4. After merging and processing the dictionary, it takes 20 hours to sort.
5. Retrieve and package and publish, and after processing more than 100G dictionaries, there are only more than 7G, which shows how careless and careless friends who made dictionaries before were. After planning, I wrote a shell script to run it unattended, and I went up every once in a while to check the server load and then check the processing progress of shell echo. Calculating, plus uploading, downloading, packaging and publishing, I spent a total of four days making this dictionary; Originally, the server only borrowed me for three days, but I didn't expect that I had misjudged the size and difficulty of the dictionary; Fortunately, it was released today. After all, it was a matter of concern. I simply executed the merger to repeat in Chinese, and did not carry out manual screening, so netizens who complained about the quality of the dictionary were very sorrowful, after all, I couldn't manually go through the dictionary of more than 100G; I can only guarantee that this is a collection of all popular dictionaries and that there are no duplicates. Processing: awk+sed+vim Processing size: about 6.55G Download address: (free) 6.55G overall compressed package: link:https://pan.baidu.com/s/12LUdmUjPAA9mL_rSkdtPtQExtraction code:Tourists, if you want to see the hidden content of this post, please ReplyTourists, if you want to see the hidden content of this post, please Reply
Run split -C 1000m dictionary name The output dictionary name will not be split into rows to split the dictionary into several small files of 1G
|
Posted on 1/19/2019 3:44:04 PM
|
|
|
Posted on 7/18/2019 5:22:27 PM
|
