sharky555
/
SM-AU
1
0
Fork 0
Code Issues 0 Pull Requests 0 Releases 0 Wiki Activity
Browse Source

detect 24bit and choose media type

master
sharky555 3 years ago
parent
353e78beae
commit
164f8ea0a2
7 changed files with 1302 additions and 1220 deletions
Unified View
  1. +3
    -3
      .gitignore
  2. +76
    -72
      README.md
  3. +815
    -751
      autoupload.py
  4. +27
    -27
      json_data/config.json.example
  5. +257
    -243
      json_data/dictionary.json
  6. +7
    -7
      requirements.txt
  7. +117
    -117
      smpy.py

+ 3
- 3
.gitignore View File

@@ -1,4 +1,4 @@
__pycache__ __pycache__
config.json config.json
*session.dat *session.dat
*.torrent *.torrent

+ 76
- 72
README.md View File

@@ -1,76 +1,76 @@
This project is **NOT** being maintained as i'd rather rewrite it from scratch. This project is **NOT** being maintained as i'd rather rewrite it from scratch.

## Overview
## Overview **JPS-AU** is a tool for automating the uploading process on jpopsuki.eu.
**JPS-AU** is a tool for automating the uploading process on jpopsuki.eu. This is intended to be used alongside BugsPy.
This is intended to be used alongside BugsPy. **Features:**

- SM Client.
**Features:** - FTP Support
- SM Client. - FLAC/MP3 Support.
- FTP Support - BugsPy .log Support.
- FLAC/MP3 Support. **Installation:**
- BugsPy .log Support. - Install requirements

```
**Installation:** pip install -r requirements.txt
- Install requirements ```
``` ## Command Usage
pip install -r requirements.txt ```
``` python autoupload.py {command_name} {ID/URL}

```
## Command Usage Command | Description | Example
``` ------------- | ------------- | -------------
python autoupload.py {command_name} {ID/URL} -d, --debug | Provides additional information on upload for debugging purposes | `python autoupload.py -d`
``` -f, --freeleech | Enables freeleech (VIP+ Userclass Requirement) | `python autoupload.py -f -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"`
Command | Description | Example -t, --tags | Add additional tags to upload, separated with comma | `python autoupload.py -t "korean, female.vocalist" -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"`
-d, --debug | Provides additional information on upload for debugging purposes | `python autoupload.py -d` -dir, --directory | Appoint directory used for torrent creation | `python autoupload.py -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"`
-f, --freeleech | Enables freeleech (VIP+ Userclass Requirement) | `python autoupload.py -f -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"` -dry, --dryrun | Carries out all actions other than the upload itself.| `python autoupload.py -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]" -dry`
-t, --tags | Add additional tags to upload, separated with comma | `python autoupload.py -t "korean, female.vocalist" -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"` -i, --imageURL | Set the torrent cover URL | `python autoupload.py -f -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3] -i "https://someimageurl.com/someimage.jpg""`
-dir, --directory | Appoint directory used for torrent creation | `python autoupload.py -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]"` ## Config.json
-dry, --dryrun | Carries out all actions other than the upload itself.| `python autoupload.py -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3]" -dry` - It's not recommended to use both local watch/download folders and ftp watch/download folders at the same time as it will result in seeding from 2 locations.
-i, --imageURL | Set the torrent cover URL | `python autoupload.py -f -dir "Z:\Music\Korean\Ego\Ego - E [2020.01.02] [EP] [WEB-MP3] -i "https://someimageurl.com/someimage.jpg""` - If generate_tracklist is set to false the script will try and find BugsPy logs within the selected log folder, this will use the comments tag to find the log. For example if 204815 is within your comments tag, it will search your log folder for 204815.log and use the contents of this log for the album description.

- Similarly to generate_tracklist, if enable_release_description is 'true' the script will create a url to the Bugs.co.kr album webpage to display the source, this can easily be edited to suit your needs within the gatherdata() function
## Config.json **credentials:**

Config | Description | Example
- It's not recommended to use both local watch/download folders and ftp watch/download folders at the same time as it will result in seeding from 2 locations. ------------- | ------------- | -------------
- If generate_tracklist is set to false the script will try and find BugsPy logs within the selected log folder, this will use the comments tag to find the log. For example if 204815 is within your comments tag, it will search your log folder for 204815.log and use the contents of this log for the album description. Username | SugoiMusic Username | Slyy
- Similarly to generate_tracklist, if enable_release_description is 'true' the script will create a url to the Bugs.co.kr album webpage to display the source, this can easily be edited to suit your needs within the gatherdata() function Password | SugoiMusic Password | Password

Passkey | SugoiMusic Passkey | Passkey
**credentials:** **local_prefs**

Config | Description | Example
Config | Description | Example ------------- | ------------- | -------------
Username | SugoiMusic Username | Slyy log_directory | directory containing BugsPy log files | `Z:/Bugs/Logs`
Password | SugoiMusic Password | Password generate_tracklist | enable tracklist generation | `true/false`
Passkey | SugoiMusic Passkey | Passkey save_tracklist | write tracklist to .log and save in log folder | `true/false`

enable_release_description | post comments tag to release description | `true/false`
**local_prefs** cover_name | name of cover with extension | `cover.jpg`

add_to_watch_folder | moves .torrent file to local watch folder | `true/false`
Config | Description | Example add_to_downloads_folder | moves torrent data to local downloads folder | `true/false`
log_directory | directory containing BugsPy log files | `Z:/Bugs/Logs` local_watch_folder | directory of local watch folder | `Z:/watch/Transmission`
generate_tracklist | enable tracklist generation | `true/false` local_downloads_folder | directory of local downloads folder | `Z:/downloads`
save_tracklist | write tracklist to .log and save in log folder | `true/false` **ftp_prefs:**
enable_release_description | post comments tag to release description | `true/false` Config | Description | Example
cover_name | name of cover with extension | `cover.jpg` ------------- | ------------- | -------------
add_to_watch_folder | moves .torrent file to local watch folder | `true/false` enable_ftp | enable ftp mode, if enabled suggested to disable local watch and downloads folders | `true/false`
add_to_downloads_folder | moves torrent data to local downloads folder | `true/false` add_to_watch_folder | transfer .torrent file to watch folder on FTP server | `true/false`
local_watch_folder | directory of local watch folder | `Z:/watch/Transmission` add_to_downloads_folder | transfer torrent data to downloads folder on FTP server | `true/false`
local_downloads_folder | directory of local downloads folder | `Z:/downloads` ftp_server | url of ftp server | haze.seedhost.eu

ftp_username | username of ftp account | slyy

ftp_password | password of ftp account | password
**ftp_prefs:** ftp_watch_folder | directory of ftp watch folder | `/downloads/watch/transmission`

ftp_downloads_folder | directory of ftp downloads folder | `/downloads`
Config | Description | Example ## Disclaimer
enable_ftp | enable ftp mode, if enabled suggested to disable local watch and downloads folders | `true/false` - The usage of this script **may be** illegal in your country. It's your own responsibility to inform yourself of Copyright Law.
add_to_watch_folder | transfer .torrent file to watch folder on FTP server | `true/false`
add_to_downloads_folder | transfer torrent data to downloads folder on FTP server | `true/false`
ftp_server | url of ftp server | haze.seedhost.eu
ftp_username | username of ftp account | slyy
ftp_password | password of ftp account | password
ftp_watch_folder | directory of ftp watch folder | `/downloads/watch/transmission`
ftp_downloads_folder | directory of ftp downloads folder | `/downloads`


## Disclaimer
- The usage of this script **may be** illegal in your country. It's your own responsibility to inform yourself of Copyright Law.

+ 815
- 751
autoupload.py
File diff suppressed because it is too large
View File


+ 27
- 27
json_data/config.json.example View File

@@ -1,27 +1,27 @@
{ {
"credentials": { "credentials": {
"username": "username", "username": "username",
"password": "password", "password": "password",
"passkey": "passkey" "passkey": "passkey"
}, },
"local_prefs": { "local_prefs": {
"log_directory": "D:/Bugs/BugsPy-master/logs", "log_directory": "D:/Bugs/BugsPy-master/logs",
"generate_tracklist": true, "generate_tracklist": true,
"save_tracklist": false, "save_tracklist": false,
"enable_release_description": false, "enable_release_description": false,
"cover_name": "cover.jpg", "cover_name": "cover.jpg",
"add_to_watch_folder": false, "add_to_watch_folder": false,
"add_to_downloads_folder": false, "add_to_downloads_folder": false,
"local_watch_folder": "C:/watch", "local_watch_folder": "C:/watch",
"local_downloads_folder": "C:/downloads" "local_downloads_folder": "C:/downloads"
}, },
"ftp_prefs": { "ftp_prefs": {
"enable_ftp": false, "enable_ftp": false,
"add_to_watch_folder": true, "add_to_watch_folder": true,
"add_to_downloads_folder": true, "add_to_downloads_folder": true,
"ftp_server": "server url", "ftp_server": "server url",
"ftp_username": "username", "ftp_username": "username",
"ftp_password": "password", "ftp_password": "password",
"ftp_watch_folder": "/downloads/watch/transmission", "ftp_watch_folder": "/downloads/watch/transmission",
"ftp_downloads_folder": "/downloads" "ftp_downloads_folder": "/downloads"
} }

+ 257
- 243
json_data/dictionary.json View File

@@ -1,244 +1,258 @@
{ {
"release_types": { "release_types": {
"Album": "Album", "Album": "Album",
"Single": "Single", "Single": "Single",
"EP": "Album", "EP": "Album",
"OST": "Album", "OST": "Album",
"싱글": "Single", "싱글": "Single",
"EP(미니)": "Album", "EP(미니)": "Album",
"정규": "Album", "정규": "Album",
"컴필레이션": "Album", "컴필레이션": "Album",
"베스트": "Album", "베스트": "Album",
"미니": "Album" "미니": "Album"
}, },
"genres": { "genres": {
"R&B": "rnb", "R&B": "rnb",
"소울": "Soul", "소울": "Soul",
"힙합": "hip.hop", "힙합": "hip.hop",
"랩": "Rap", "랩": "Rap",
"영화": "Movie", "영화": "Movie",
"로맨스": "Romance", "로맨스": "Romance",
"드라마": "OST", "드라마": "OST",
"TV 드라마": "OST", "TV 드라마": "OST",
"애니메이션": "anime", "애니메이션": "anime",
"인디": "Indie", "인디": "Indie",
"인디힙합": "Indie Hip-Hop", "인디힙합": "Indie Hip-Hop",
"재즈 힙합": "Jazz-Hop", "재즈 힙합": "Jazz-Hop",
"댄스 팝": "Dance", "댄스 팝": "Dance",
"발라드": "Ballad", "발라드": "Ballad",
"댄스": "Dance", "댄스": "Dance",
"포크 팝": "Folk", "포크 팝": "Folk",
"팝": "Pop", "팝": "Pop",
"팝 락": "Pop.Rock", "팝 락": "Pop.Rock",
"인디 락": "Indie.Rock", "인디 락": "Indie.Rock",
"락": "Rock", "락": "Rock",
"메탈": "Metal", "메탈": "Metal",
"인디 팝": "Indie.Pop", "인디 팝": "Indie.Pop",
"일렉트로닉": "Electronic", "일렉트로닉": "Electronic",
"일렉트로닉 팝": "Electro", "일렉트로닉 팝": "Electro",
"인디일렉트로닉": "Indie.Electronic", "인디일렉트로닉": "Indie.Electronic",
"신스 팝": "Synth-Pop", "신스 팝": "Synth-Pop",
"J-POP": "J-Pop", "J-POP": "J-Pop",
"재즈": "Jazz", "재즈": "Jazz",
"성인가요": "Trot", "성인가요": "Trot",
"월드뮤직": "World Music", "월드뮤직": "World Music",
"국악": "Traditional", "국악": "Traditional",
"종교": "Religious", "종교": "Religious",
"CCM": "CCM", "CCM": "CCM",
"어린이": "Child", "어린이": "Child",
"태교": "Taegyo", "태교": "Taegyo",
"캐롤": "Christmas", "캐롤": "Christmas",
"트랩": "Trap", "트랩": "Trap",
"얼터너티브 팝": "Alternative.Pop", "얼터너티브 팝": "Alternative.Pop",
"얼터너티브": "Alternative", "얼터너티브": "Alternative",
"뉴에이지": "New Age", "뉴에이지": "New Age",
"켈틱": "Celtic", "켈틱": "Celtic",
"켈틱 퓨전": "Celtic.Fusion", "켈틱 퓨전": "Celtic.Fusion",
"퓨전": "Fusion", "퓨전": "Fusion",
"에스닉 퓨전": "Ethnic.Fusion", "에스닉 퓨전": "Ethnic.Fusion",
"레게": "Reggae", "레게": "Reggae",
"댄스홀": "Dancehall", "댄스홀": "Dancehall",
"하우스": "House", "하우스": "House",
"트로트": "Trot", "트로트": "Trot",
"얼터너티브 락": "Alternative.Rock", "얼터너티브 락": "Alternative.Rock",
"덥": "Dub", "덥": "Dub",
"싸이키델릭": "Psychedelic", "싸이키델릭": "Psychedelic",
"인스트루멘탈 힙합": "Instrumental.Hip-Hop", "인스트루멘탈 힙합": "Instrumental.Hip-Hop",
"인스트루멘탈": "Instrumental", "인스트루멘탈": "Instrumental",
"클래식": "Classic", "클래식": "Classic",
"컨트리": "Country", "컨트리": "Country",
"종교음악": "Religious", "종교음악": "Religious",
"전통음악": "Traditional", "전통음악": "Traditional",
"블루스": "Blues", "블루스": "Blues",
"라틴": "Latin", "라틴": "Latin",
"기타": "Other", "기타": "Other",
"기능성음악": "Functional", "기능성음악": "Functional",
"인디포크": "indie.folk", "인디포크": "indie.folk",
"포크": "Folk", "포크": "Folk",
"어쿠스틱": "Acoustic", "어쿠스틱": "Acoustic",
"Hip-Hop": "hip.hop" "Hip-Hop": "hip.hop"
}, },
"artist": { "artist": {
"오아": "OA", "오아": "OA",
"이고": "Ego", "이고": "Ego",
"ハルカトミユキ": null, "ハルカトミユキ": null,
"琴音": null, "琴音": null,
"下村陽子 × suis from ヨルシカ": null, "下村陽子 × suis from ヨルシカ": null,
"川島ケイジ": null, "川島ケイジ": null,
"裸体": null, "裸体": null,
"空音": null, "空音": null,
"さかいゆう": null, "さかいゆう": null,
"美波": null, "美波": null,
"アルカラ": null, "アルカラ": null,
"윤상": null, "윤상": null,
"ブレッド & バター": null, "ブレッド & バター": null,
"Official髭男dism": null, "Official髭男dism": null,
"優里": null, "優里": null,
"サニーデイ・サービス": null, "サニーデイ・サービス": null,
"ずっと真夜中でいいのに。": null, "ずっと真夜中でいいのに。": null,
"やなぎなぎ": null, "やなぎなぎ": null,
"米津玄師": null, "米津玄師": null,
"梶浦由記": null, "梶浦由記": null,
"澁谷逆太郎": null, "澁谷逆太郎": null,
"ポルカドットスティングレイ": null, "ポルカドットスティングレイ": null,
"김트와친구들": null, "김트와친구들": null,
"安斉かれん": null, "安斉かれん": null,
"坂口有望": null, "坂口有望": null,
"空想委員会": null, "空想委員会": null,
"ヨルシカ": null, "ヨルシカ": null,
"向井太一": null, "向井太一": null,
"ペンギンラッシュ": null, "ペンギンラッシュ": null,
"黒子首": null, "黒子首": null,
"中島みゆき": null, "中島みゆき": null,
"ハリィさんとスイカくらぶ": null, "ハリィさんとスイカくらぶ": null,
"堀込高樹": null, "堀込高樹": null,
"堀込泰行": null, "堀込泰行": null,
"スピラ・スピカ": null, "スピラ・スピカ": null,
"17歳とベルリンの壁": null, "17歳とベルリンの壁": null,
"天野月": null, "天野月": null,
"ソールドシュガー": null, "ソールドシュガー": null,
"ナンカノユメ": null, "ナンカノユメ": null,
"ルルルルズ": null, "ルルルルズ": null,
"東京事変": null, "東京事変": null,
"藍井エイル": null, "藍井エイル": null,
"阿部真央": null, "阿部真央": null,
"赤いくらげ": null, "赤いくらげ": null,
"週末CITY PLAY BOYZ": null, "週末CITY PLAY BOYZ": null,
"林 浩司": null, "林 浩司": null,
"蒼山幸子": null, "蒼山幸子": null,
"フラスコテーション": null, "フラスコテーション": null,
"ゑんら": null, "ゑんら": null,
"ハンブレッダーズ": null, "ハンブレッダーズ": null,
"鈴木このみ": null, "鈴木このみ": null,
"みゆな": null, "みゆな": null,
"ビッケブランカ": null, "ビッケブランカ": null,
"めありー": null, "めありー": null,
"キタニタツヤ": null, "キタニタツヤ": null,
"イロメガネ": null, "イロメガネ": null,
"ヤユヨ": null, "ヤユヨ": null,
"ピロカルピン": null, "ピロカルピン": null,
"ツユ": null, "ツユ": null,
"リリー楽綺団": null, "リリー楽綺団": null,
"山崎ハコ": null, "山崎ハコ": null,
"いきものがかり": null, "いきものがかり": null,
"はるまきごはん": null, "はるまきごはん": null,
"おくみずき": null, "おくみずき": null,
"渣泥": null, "渣泥": null,
"竹渕慶": null, "竹渕慶": null,
"早見沙織": null, "早見沙織": null,
"倖田來未": null, "倖田來未": null,
"世武裕子": null, "世武裕子": null,
"ラブリーサマーちゃん": null, "ラブリーサマーちゃん": null,
"SUPER☆GiRLS": null, "SUPER☆GiRLS": null,
"österreich": null, "österreich": null,
"フレデリック": null, "フレデリック": null,
"ズーカラデル": null, "ズーカラデル": null,
"神山羊": null, "神山羊": null,
"太田ひな": null, "太田ひな": null,
"ヤバイTシャツ屋さん": null, "ヤバイTシャツ屋さん": null,
"當山みれい": null, "當山みれい": null,
"大森靖子": null, "大森靖子": null,
"大原櫻子": null, "大原櫻子": null,
"東京スカパラダイスオーケストラ": null, "東京スカパラダイスオーケストラ": null,
"三月のパンタシア": null, "三月のパンタシア": null,
"雨のパレード": null, "雨のパレード": null,
"川崎鷹也": null, "川崎鷹也": null,
"中島 美嘉": null, "中島 美嘉": null,
"加藤ミリヤ": null, "加藤ミリヤ": null,
"りぶ": null, "りぶ": null,
"雨ニマケテモ": null, "雨ニマケテモ": null,
"三浦大知": null, "三浦大知": null,
"コブクロ": null, "コブクロ": null,
"ももいろクローバーZ": null, "ももいろクローバーZ": null,
"手嶌葵": null, "手嶌葵": null,
"Nao☆": null, "Nao☆": null,
"尾崎裕哉": null, "尾崎裕哉": null,
"マーティ・フリードマン": null, "マーティ・フリードマン": null,
"幾田りら": null, "幾田りら": null,
"山本彩": null, "山本彩": null,
"ビッケブランカ VS 岡崎体育": null, "ビッケブランカ VS 岡崎体育": null,
"まるりとりゅうが": null, "まるりとりゅうが": null,
"藤原さくら": null, "藤原さくら": null,
"藤井風": null, "藤井風": null,
"sicboy": "", "sicboy": "",
"LUCA & haruka nakamura": "arca", "LUCA & haruka nakamura": "arca",
"伊沢麻未": null, "伊沢麻未": null,
"マカロニえんぴつ": null, "マカロニえんぴつ": null,
"チャラン・ポ・ランタン": null, "チャラン・ポ・ランタン": null,
"鈴木瑛美子": null, "鈴木瑛美子": null,
"神はサイコロを振らない": null, "神はサイコロを振らない": null,
"宇野実彩子": "AAA", "宇野実彩子": "AAA",
"ウルトラタワー": null, "ウルトラタワー": null,
"空白ごっこ": null, "空白ごっこ": null,
"Cö shu Nie": null, "Cö shu Nie": null,
"くるり": null, "くるり": null,
"流線形 & 一十三十一": null, "流線形 & 一十三十一": null,
"清水翔太": null, "清水翔太": null,
"あれくん": null, "あれくん": null,
"秋山黄色": null, "秋山黄色": null,
"웬디": "WENDY", "웬디": "WENDY",
"瀧川ありさ": null, "瀧川ありさ": null,
"キリンジ": null, "キリンジ": null,
"ユアネス": null, "ユアネス": null,
"クレナズム": null, "クレナズム": null,
"H△G": null, "H△G": null,
"電音部": null, "電音部": null,
"武藤彩未": null, "武藤彩未": null,
"中島美嘉": null, "中島美嘉": null,
"雫": null, "雫": null,
"坂本真綾": null, "坂本真綾": null,
"たかやん": null, "たかやん": null,
"布袋寅泰": null, "布袋寅泰": null,
"アイラヴミー": null, "アイラヴミー": null,
"ナナヲアカリ": null, "ナナヲアカリ": null,
"福山雅治": null, "福山雅治": null,
"Jacob&よみぃ": null, "Jacob&よみぃ": null,
"クミコ": null, "クミコ": null,
"リュックと添い寝ごはん": null, "リュックと添い寝ごはん": null,
"眉村ちあき": null, "眉村ちあき": null,
"ちゃんみな & SKY-HI": null, "ちゃんみな & SKY-HI": null,
"関口シンゴ": null, "関口シンゴ": null,
"角巻わため": null, "角巻わため": null,
"Snail’s House": null, "Snail’s House": null,
"ロザリーナ": null, "ロザリーナ": null,
"ニノミヤユイ": null, "ニノミヤユイ": null,
"シド": null, "シド": null,
"森内寛樹": null, "森内寛樹": null,
"TK from 凛として時雨": null, "TK from 凛として時雨": null,
"スダンナユズユリー": null, "スダンナユズユリー": null,
"ヤなことそっとミュート": null, "ヤなことそっとミュート": null,
"宇徳敬子": null, "宇徳敬子": null,
"Heavenstamp & ハハノシキュウ": null, "Heavenstamp & ハハノシキュウ": null,
"비": null, "비": null,
"ぼっちぼろまる": null, "ぼっちぼろまる": null,
"さなり": null, "さなり": null,
"家入レオ": null, "家入レオ": null,
"凛として時雨": null, "凛として時雨": null,
"緑黄色社会": null, "緑黄色社会": null,
"あるゆえ": null, "あるゆえ": null,
"宮下 遊": null, "宮下 遊": null,
"夢乃ゆき": null, "夢乃ゆき": null,
"女王蜂": null, "女王蜂": null,
"夜の本気ダンス": null "夜の本気ダンス": null,
} "藤末樹": null,
"崎山蒼志": null,
"草野華余子": null,
"マット・キャブ & MIYACHI": null,
"Maika Loubté": null,
"青山みつ紀": null,
"アイナ・ジ・エンド": null,
"須田景凪": null,
"あっこゴリラ": null,
"小林香織": null,
"L'Arc～en～Ciel": null,
"星街すいせい": null,
"高槻かなこ": null,
"Leeu & こゆき": null
}
} }

+ 7
- 7
requirements.txt View File

@@ -1,7 +1,7 @@
bs4 == 0.0.1 bs4 == 0.0.1
langdetect == 1.0.7 langdetect == 1.0.7
mutagen == 1.42.0 mutagen == 1.42.0
requests == 2.22.0 requests == 2.22.0
torf == 2.1.0 torf == 2.1.0
tqdm == 4.39.0 tqdm == 4.39.0
html5lib html5lib

+ 117
- 117
smpy.py View File

@@ -1,117 +1,117 @@
import os import os
import pickle import pickle
import datetime import datetime
from urllib.parse import urlparse from urllib.parse import urlparse
import requests import requests
class MyLoginSession:
def __init__(self,
class MyLoginSession: loginUrl,
def __init__(self, loginData,
loginUrl, loginTestUrl,
loginData, loginTestString,
loginTestUrl, sessionFileAppendix='_session.dat',
loginTestString, maxSessionTimeSeconds=30 * 60,
sessionFileAppendix='_session.dat', proxies=None,
maxSessionTimeSeconds=30 * 60, userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1',
proxies=None, debug=False,
userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1', forceLogin=False,
debug=False, **kwargs):
forceLogin=False, """
**kwargs): save some information needed to login the session
""" you'll have to provide 'loginTestString' which will be looked for in the
save some information needed to login the session responses html to make sure, you've properly been logged in
'proxies' is of format { 'https' : 'https://user:pass@server:port', 'http' : ...
you'll have to provide 'loginTestString' which will be looked for in the 'loginData' will be sent as post data (dictionary of id : value).
responses html to make sure, you've properly been logged in 'maxSessionTimeSeconds' will be used to determine when to re-login.
"""
'proxies' is of format { 'https' : 'https://user:pass@server:port', 'http' : ... urlData = urlparse(loginUrl)
'loginData' will be sent as post data (dictionary of id : value). self.proxies = proxies
'maxSessionTimeSeconds' will be used to determine when to re-login. self.loginData = loginData
""" self.loginUrl = loginUrl
urlData = urlparse(loginUrl) self.loginTestUrl = loginTestUrl
self.maxSessionTime = maxSessionTimeSeconds
self.proxies = proxies self.sessionFile = urlData.netloc + sessionFileAppendix
self.loginData = loginData self.userAgent = userAgent
self.loginUrl = loginUrl self.loginTestString = loginTestString
self.loginTestUrl = loginTestUrl self.debug = debug
self.maxSessionTime = maxSessionTimeSeconds self.login(forceLogin, **kwargs)
self.sessionFile = urlData.netloc + sessionFileAppendix def modification_date(self, filename):
self.userAgent = userAgent """
self.loginTestString = loginTestString return last file modification date as datetime object
self.debug = debug """
t = os.path.getmtime(filename)
self.login(forceLogin, **kwargs) return datetime.datetime.fromtimestamp(t)
def login(self, forceLogin=False, **kwargs):
def modification_date(self, filename): """
""" login to a session. Try to read last saved session from cache file. If this fails
return last file modification date as datetime object do proper login. If the last cache access was too old, also perform a proper login.
""" Always updates session cache file.
t = os.path.getmtime(filename) """
return datetime.datetime.fromtimestamp(t) wasReadFromCache = False
if self.debug:
def login(self, forceLogin=False, **kwargs): print('loading or generating session...')
""" if os.path.exists(self.sessionFile) and not forceLogin:
login to a session. Try to read last saved session from cache file. If this fails time = self.modification_date(self.sessionFile)
do proper login. If the last cache access was too old, also perform a proper login. # only load if file less than 30 minutes old
Always updates session cache file. lastModification = (datetime.datetime.now() - time).seconds
""" if lastModification < self.maxSessionTime:
wasReadFromCache = False with open(self.sessionFile, "rb") as f:
if self.debug: self.session = pickle.load(f)
print('loading or generating session...') wasReadFromCache = True
if os.path.exists(self.sessionFile) and not forceLogin: if self.debug:
time = self.modification_date(self.sessionFile) print("loaded session from cache (last access %ds ago) "
% lastModification)
# only load if file less than 30 minutes old if not wasReadFromCache:
lastModification = (datetime.datetime.now() - time).seconds self.session = requests.Session()
if lastModification < self.maxSessionTime: self.session.headers.update({'user-agent': self.userAgent})
with open(self.sessionFile, "rb") as f: res = self.session.post(self.loginUrl, data=self.loginData,
self.session = pickle.load(f) proxies=self.proxies, **kwargs)
wasReadFromCache = True if self.debug:
if self.debug: print('created new session with login')
print("loaded session from cache (last access %ds ago) " self.saveSessionToCache()
% lastModification) # test login
if not wasReadFromCache: res = self.session.get(self.loginTestUrl)
self.session = requests.Session() if res.text.lower().find(self.loginTestString.lower()) < 0:
self.session.headers.update({'user-agent': self.userAgent}) if self.debug:
res = self.session.post(self.loginUrl, data=self.loginData, print(res.text)
proxies=self.proxies, **kwargs) raise Exception("could not log into provided site '%s'"
" (did not find successful login string)"
if self.debug: % self.loginUrl)
print('created new session with login') def saveSessionToCache(self):
self.saveSessionToCache() """
save session to a cache file
# test login """
res = self.session.get(self.loginTestUrl) # always save (to update timeout)
if res.text.lower().find(self.loginTestString.lower()) < 0: with open(self.sessionFile, "wb") as f:
if self.debug: pickle.dump(self.session, f)
print(res.text) if self.debug:
raise Exception("could not log into provided site '%s'" print('updated session cache-file %s' % self.sessionFile)
" (did not find successful login string)" def retrieveContent(self, url, method="get", postData=None, postDataFiles=None, **kwargs):
% self.loginUrl) """
return the content of the url with respect to the session.
def saveSessionToCache(self): If 'method' is not 'get', the url will be called with 'postData'
""" as a post request.
save session to a cache file """
""" if method == 'get':
# always save (to update timeout) res = self.session.get(url, proxies=self.proxies, **kwargs)
with open(self.sessionFile, "wb") as f: else:
pickle.dump(self.session, f) res = self.session.post(url, data=postData, proxies=self.proxies, files=postDataFiles, **kwargs)
if self.debug: # the session has been updated on the server, so also update in cache
print('updated session cache-file %s' % self.sessionFile) self.saveSessionToCache()
return res
def retrieveContent(self, url, method="get", postData=None, postDataFiles=None, **kwargs):
"""
return the content of the url with respect to the session.
If 'method' is not 'get', the url will be called with 'postData'
as a post request.
"""
if method == 'get':
res = self.session.get(url, proxies=self.proxies, **kwargs)
else:
res = self.session.post(url, data=postData, proxies=self.proxies, files=postDataFiles, **kwargs)
# the session has been updated on the server, so also update in cache
self.saveSessionToCache()
return res

Write Preview
||||||
​x
 
​
000:0
Loading…
Cancel
Save