Merging all Articles

I scheduled this notebook for 3 times a week. Then merged every dataframe into one big dataset called 'combined.csv'. Before merging datasets, each data is cleaned in their individual notebooks. All dates were converted into datetime. Missing descriptions were replaced with 'No Description', nans in the category columns were filled. After merging, the dataframe is also cleaned to drop duplicates, drop excess columns that were created when merged.

import os import pandas as pd import numpy as np from pathlib import Path from os import listdir import glob from datetime import datetime import os.path

os.listdir('/datasets/webscrape/')

os.chdir(os.path.dirname('/datasets/webscrape/')) extension = 'csv' all_filenames = [i for i in glob.glob('*.{}'.format(extension))] for filename in all_filenames: if filename != 'send.csv' and 'new_articles.csv': combined = pd.concat([pd.read_csv(f) for f in all_filenames], ignore_index=True) os.chdir(os.path.dirname('/datasets/webscrape/'))

combined

idfloat64

0.0 - 256.0

Titleobject

Freestyle wrestlers lead world ranking0.4%

1979 others99.6%

Missing0%

16869

248.0

CABINET DECIDES TO BAN RAW COAL CONSUMPTION IN 2019

16870

249.0

TRQ: “OYU TOLGOI” COPPER PRODUCTION AND GOLD PRODUCTION WILL INCREASE BY 340 PERCENT AND 150 PERCENT RESPECTIVELY IN 2018-2025

16871

250.0

FOREIGN MINISTER HIGHLIGHTS ECONOMIC RECOVERY AT BCM MONTHLY MEETING

16872

251.0

JEROME POWELL: THE FEDERAL RESERVE RAISING U.S. INTEREST RATES FOUR TIMES THIS YEAR

19185

251.0

JEROME POWELL: THE FEDERAL RESERVE RAISING U.S. INTEREST RATES FOUR TIMES THIS YEAR

15587

251.0

JEROME POWELL: THE FEDERAL RESERVE RAISING U.S. INTEREST RATES FOUR TIMES THIS YEAR

19184

250.0

FOREIGN MINISTER HIGHLIGHTS ECONOMIC RECOVERY AT BCM MONTHLY MEETING

15586

250.0

FOREIGN MINISTER HIGHLIGHTS ECONOMIC RECOVERY AT BCM MONTHLY MEETING

19183

249.0

TRQ: “OYU TOLGOI” COPPER PRODUCTION AND GOLD PRODUCTION WILL INCREASE BY 340 PERCENT AND 150 PERCENT RESPECTIVELY IN 2018-2025

19182

248.0

CABINET DECIDES TO BAN RAW COAL CONSUMPTION IN 2019

combined = combined.drop_duplicates(keep='last', subset=['Title'])

combined

idfloat64

0.0 - 256.0

Titleobject

iDAX Exchange receives Virtual Asset Service Provider license.0.1%

1979 others99.9%

Missing0.1%

297

nan

iDAX Exchange receives Virtual Asset Service Provider license.

298

nan

Michidmaa Budsuren named as CEO of MEC partners (ArdShop.mn)

299

nan

Odbayar O. named as Chairman of Ard Insurance JSC

300

nan

Ganzorig Ch. appointed as Chairman of Ard Credit NBFI JSC

301

nan

The 3rd Alumni Economic Forum Opens in Arxan in Inner Mongolia - PR Newswire

302

nan

The 3rd Alumni Economic Forum Opens in Arxan in Inner Mongolia - PR Newswire APAC

303

nan

Western Inner Mongolia Series | Episode 9: Demoiselle hermit - CGTN

304

nan

The £2.2 billion offer from Rio Tinto for the copper site in Mongolia was rejected.

305

nan

Rio 'disappointed' in Turquoise Hill decision - Mining Weekly

306

nan

Turquoise Hill shares fall after Quebec miner rejects Rio Tinto's $2.7-billion ... - Yahoo Finance

combined = combined.drop(columns=['id', 'Unnamed: 0', 'X', 'y', 'prob'])

combined['Description'] = combined['Description'].fillna('No description')

combined = combined.dropna(axis=0, subset=['Date'])

combined['Date'] = combined['Date'].apply(lambda x: datetime.strptime(x,'%Y-%m-%d'))

combined['Category'] = combined['Category'].replace(['Economic', 'Social','Ярилцлага','Аялал','Ulaanbaatar'],['Economics', 'Society','Interview','Travel','Domestic'])

combined['Category'][102:137] = combined['Category'][102:137].fillna('International')

/tmp/ipykernel_73/2806525063.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  combined['Category'][102:137] = combined['Category'][102:137].fillna('International')

combined['Category'] = combined['Category'].fillna('Finance')

combined = combined.dropna(axis=1)

combined = combined.reset_index()

combined = combined.drop(columns = ['index'])

combined

Titleobject

iDAX Exchange receives Virtual Asset Service Provider license.0.1%

Michidmaa Budsuren named as CEO of MEC partners (ArdShop.mn)0.1%

1978 others99.9%

Datedatetime64[ns]

2016-08-17 00:00:00 - 2022-10-31 00:00:00

1960

P.Orkhon triumphs at Asian Wrestling Championships

2018-03-29 00:00:00

1961

Lessons from UBIFF-2022 should stick around

2022-10-30 00:00:00

1962

Z.Tumenjargal and B.Bayartsengel collaborate on new exhibition

2022-10-26 00:00:00

1963

‘B-Atelier-209’ art show unveiled

2022-10-14 00:00:00

1964

‘Art has become something only for a certain group of people’

2022-10-07 00:00:00

1965

Students document social issues in photos

2022-09-17 00:00:00

1966

Viewing rural life from children’s perspective

2022-06-03 00:00:00

1967

‘Fantastic Creatures’ art show on display

2022-05-20 00:00:00

1968

Traverse nature at ‘Anir’ art show

2022-05-07 00:00:00

1969

Postmodern art show highlights social issues

2022-04-29 00:00:00

#combined.to_csv( "/work/merged.csv", index=False, encoding='utf-8-sig')

combined.to_csv('/datasets/webscrape/combined.csv', index=False)

Find Old/New Rows

When I though I had a good amount of articles to train (804 articles) I manually inserted 1's an 0's into a target column of that dataset which shows which articles I would want to send out to people and which ones I wouldn't and named that dataset 'combined_csv'.

The dataset 'combined_csv' will become my old dataset, and then the newly merged dataset would just be called 'combined'. Then I loop through the files and get the recent file which would contain old and new articles. Then I would get the length of the old dataset and slice the newly combined dataset to get a dataframe called 'new_articles'. Then I save the new_article to a csv, concat the dataframe to the old dataset for when the next new articles merge it would become the old dataset.

old_combined = pd.read_csv("/datasets/webscrape/combined_csv")

#old_combined = old_combined.dropna(axis= 1)

#old_combined = old_combined.drop(columns = ['Unnamed: 0.1'])

old_combined

Titleobject

THE HU PLANS TO HOST A VIRTUAL CONCERT IN THE METAVERSE IN PARTNERSHIP WITH ARD0.1%

INVESTOR NATION – MONGOLIA’S FIRST OPEN-END MUTUAL FUND WAS ESTABLISHED AND IS SET TO MAKE AN INITIAL PUBLIC OFFERING0.1%

802 others99.8%

Dateobject

2022-08-084.1%

2022-08-043.9%

345 others92%

THE HU PLANS TO HOST A VIRTUAL CONCERT IN THE METAVERSE IN PARTNERSHIP WITH ARD

2022-07-01

INVESTOR NATION – MONGOLIA’S FIRST OPEN-END MUTUAL FUND WAS ESTABLISHED AND IS SET TO MAKE AN INITIAL PUBLIC OFFERING

2022-05-16

BULGAN UNDRAGA JSC CHANGES NAME TO CRYPTO NATION JSC

2022-03-18

ARD FINANCIAL GROUP COMMITS TO WORKING WITH SMEs

2022-02-24

ARD FINANCIAL GROUP TO ALLOCATE DIVIDENDS TO SHAREHOLDERS

2022-02-16

Ard Financial Group highlighted as Technology Investor Entrepreneur of the Year

2021-12-29

MONGOLIA ADOPTS LAW ON VIRTUAL ASSET SERVICE PROVIDERS

2021-12-16

ARD STARTS PARTNERSHIP WITH ZEUS ESPORTS TEAM

2021-07-08

MONGOLIA’S FIRST PUBLIC MEDIA GROUP

2021-07-02

Trade between Mongolia and China reached USD 10 billion in 2021

2022-06-22

os.chdir(os.path.dirname('/datasets/webscrape/')) extension = 'csv' all_filenames = [i for i in glob.glob('*.{}'.format(extension))] max_file = max(all_filenames, key=os.path.getctime) print(max_file)

combined.csv

new_combined = pd.read_csv(max_file)

new_combined

Titleobject

iDAX Exchange receives Virtual Asset Service Provider license.0.1%

Michidmaa Budsuren named as CEO of MEC partners (ArdShop.mn)0.1%

1978 others99.9%

Dateobject

2022-08-161.9%

2022-08-181.8%

424 others96.3%

iDAX Exchange receives Virtual Asset Service Provider license.

2022-09-23

Michidmaa Budsuren named as CEO of MEC partners (ArdShop.mn)

2022-08-17

Odbayar O. named as Chairman of Ard Insurance JSC

2022-07-22

Ganzorig Ch. appointed as Chairman of Ard Credit NBFI JSC

2022-06-20

The 3rd Alumni Economic Forum Opens in Arxan in Inner Mongolia - PR Newswire

2022-08-16

The 3rd Alumni Economic Forum Opens in Arxan in Inner Mongolia - PR Newswire APAC

2022-08-16

Western Inner Mongolia Series | Episode 9: Demoiselle hermit - CGTN

2022-08-16

The £2.2 billion offer from Rio Tinto for the copper site in Mongolia was rejected.

2022-08-16

Rio 'disappointed' in Turquoise Hill decision - Mining Weekly

2022-08-16

Turquoise Hill shares fall after Quebec miner rejects Rio Tinto's $2.7-billion ... - Yahoo Finance

2022-08-16

old_combined

Titleobject

THE HU PLANS TO HOST A VIRTUAL CONCERT IN THE METAVERSE IN PARTNERSHIP WITH ARD0.1%

INVESTOR NATION – MONGOLIA’S FIRST OPEN-END MUTUAL FUND WAS ESTABLISHED AND IS SET TO MAKE AN INITIAL PUBLIC OFFERING0.1%

802 others99.8%

Dateobject

2022-08-084.1%

2022-08-043.9%

345 others92%

THE HU PLANS TO HOST A VIRTUAL CONCERT IN THE METAVERSE IN PARTNERSHIP WITH ARD

2022-07-01

INVESTOR NATION – MONGOLIA’S FIRST OPEN-END MUTUAL FUND WAS ESTABLISHED AND IS SET TO MAKE AN INITIAL PUBLIC OFFERING

2022-05-16

BULGAN UNDRAGA JSC CHANGES NAME TO CRYPTO NATION JSC

2022-03-18

ARD FINANCIAL GROUP COMMITS TO WORKING WITH SMEs

2022-02-24

ARD FINANCIAL GROUP TO ALLOCATE DIVIDENDS TO SHAREHOLDERS

2022-02-16

Ard Financial Group highlighted as Technology Investor Entrepreneur of the Year

2021-12-29

MONGOLIA ADOPTS LAW ON VIRTUAL ASSET SERVICE PROVIDERS

2021-12-16

ARD STARTS PARTNERSHIP WITH ZEUS ESPORTS TEAM

2021-07-08

MONGOLIA’S FIRST PUBLIC MEDIA GROUP

2021-07-02

Trade between Mongolia and China reached USD 10 billion in 2021

2022-06-22

new_articles = new_combined[len(old_combined):]

new_articles = new_articles.reset_index(drop=True)

new_articles

Titleobject

Prime Minister of Mongolia Oyun-Erdene Wants to Get the Country Out of Pandemic Slowdown0.1%

Backlash over Development Bank of Mongolia Exposes Social and Political Tensions0.1%

1174 others99.8%

Dateobject

2022-10-111.4%

2022-10-211.3%

346 others97.4%

Prime Minister of Mongolia Oyun-Erdene Wants to Get the Country Out of Pandemic Slowdown

2022-04-29

Backlash over Development Bank of Mongolia Exposes Social and Political Tensions

2022-04-27

What You Need To Know About Mongolia and What's On the Horizon in H2 2022

2022-05-25

Mongolia Starts Austerity Measures

2022-05-09

Mongolia Cuts Red Tape By Breaking Up GASI

2022-10-30

Cracks Become More Visible Within the Ruling Party

2022-10-24

Government Offers Tax Breaks For Tech Startups & Financial Sector

2022-10-21

Germany Gets Interested In Mongolia’s Critical Minerals

2022-10-17

A Road Connecting China and Europe Probed For Corruption

2022-10-17

Chinese Steelmakers Turn Away From Coking Coal Risking Mongolia’s Key Earning

2022-10-03

new_articles.to_csv('/datasets/webscrape/new_articles.csv', index=False)

old_combined = pd.concat([old_combined, new_articles])

old_combined.to_csv('/datasets/webscrape/combined_csv.csv')

.css-ytumd6{-webkit-text-decoration:none;text-decoration:none;}Merging all Articles

Find Old/New Rows

Merging all Articles