Ubuntu-fr

Feral

Extraction de données => CSV [RESOLU]

Re-Bonjour,

Merci à tous ceux qui m'ont déjà aidé sur mon petit projet.
Je suis en train de bosser sur la constitution d'une base de donnée de vétérinaires Australiens.
J'ai un fichier extrait d'une carte google qui se charge sur un site drupal.

Voici un extrait du fichier:
(J'ai découpé en 4 morceaux pour plus de lisibilité mais normalement c'est tout collé et tout est en 1 seule ligne.

{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003ENowra Veterinary Hospital\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E103 Kinghorne Street\nNOWRA NSW 2541\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E02 4421 5370\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Alison Havadjia, Dr Andrew Havadjia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-website\u0022\u003E    \u003Cspan class=\u0022views-label views-label-website\u0022\u003EWebsite: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.nowravets.com.au\u0022\u003Ewww.nowravets.com.au\u003C\/a\u003E\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-34.8803","longitude":"150.602","title":"","markername":"","offset":"","":"\u003Cb\u003ENowra Veterinary Hospital\u003C\/b\u003E","opts":{"title":""}},

{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EClaws and Paws Family Veterinary Hospital\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E44 Vittoria Street\nWEST BATHURST NSW 2795\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E02 6332 9096\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Caitlin Anderson\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-website\u0022\u003E    \u003Cspan class=\u0022views-label views-label-website\u0022\u003EWebsite: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.clawsandpaws.net.au\u0022\u003Ewww.clawsandpaws.net.au\u003C\/a\u003E\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-33.5418","longitude":"149.553","title":"","markername":"","offset":"","":"\u003Cb\u003EClaws and Paws Family Veterinary Hospital\u003C\/b\u003E","opts":{"title":""}},

{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EGray Street Veterinary Surgery\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003EGray Street\nEMERALD QLD 4720\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E07 4982 4868\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Steven Bliss\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-23.5333","longitude":"148.15299999999999","title":"","markername":"","offset":"","":"\u003Cb\u003EGray Street Veterinary Surgery\u003C\/b\u003E","opts":{"title":""}},

{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EGayndah Veterinary Surgery\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E2-4 Bridge Street\nGAYNDAH QLD 4625\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E07 4161 1404\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Nathan Hitchcock, Dr Sara Kahler\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-website\u0022\u003E    \u003Cspan class=\u0022views-label views-label-website\u0022\u003EWebsite: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.localvet.com.au\/Gayndah\/\u0022\u003Ewww.localvet.com.au\/Gayndah\/\u003C\/a\u003E\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-25.6199","longitude":"151.61099999999999","title":"","markername":"","offset":"","":"\u003Cb\u003EGayndah Veterinary Surgery\u003C\/b\u003E","opts":{"title":""}},

Donc de cette grande ligne j'aimerai bien générer un CSV de 5 champs (nom;adresse;tel;liste_des_docteurs;site_web).
Les ';' peuvent être remplacés par ce que vous voulez bien sûr.

Exemple sur dernier paragraphe, une ligne du CSV donnerait donc:

Gayndah Veterinary Surgery;2-4 Bridge Street GAYNDAH QLD 4625;07 4161 1404;Dr Nathan Hitchcock, Dr Sara Kahler;www.localvet.com.au/Gayndah/

Notez que pour certaines cliniques quelques champs sont vides.

Bref mon analyse me dit que chaque entitée commençant par '{"text":' ... et finissant par '}},' me donne les info que j'aimerai bien collecter par clinique comme ceci:

class=\u0022field-content\u0022\u003E\u003Cb\u003E*******NOM_DE_CLINIQUE********\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv 
class=\u0022field-content\u0022\u003E*******ADRESSE_DE_CLINIQUE********\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv 
class=\u0022field-content\u0022\u003E*******TELEPHONE_DE_CLINIQUE********\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv 
class=\u0022field-content\u0022\u003E*******MEMBRES_DE_CLINIQUE********\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv 
class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.localvet.com.au\/Gayndah\/\u0022\u003E*******SITE_WEB_DE_CLINIQUE********\u003C\/a\u003E\u003C\/span\u003E

Quelqu'un aurait-il une idée de comment je pourrais extraire ceci?

Merci pour votre aide!

Dernière modification par Feral (Le 04/04/2016, à 13:31)

αjet

Re : Extraction de données => CSV [RESOLU]

Avec quel language compte tu proceder ?

En ce qui concerne tes données sources, elles sont formattees en JSON, donc tu devrais parser ces données avec un lib json, ca te permettra deja de separer les differentes champs : text, latitude, longitude, title, markername, etc.

Le 1er champ contient lui un fragment de page html on dirait. Certains caracteres speciaux (tels que les balises < et >) sont encodees. Pour recuperer le contenu texte, tu devrais cette fois ci songer a decoder le texte (ex \u003C => <) et utiliser un parser pour html. Par exemple beautifulsoup: http://www.pythonforbeginners.com/beautifulsoup/.

Pour ce qui est de ton format de sortie, pourquoi specifiquement du csv ?
Si c'est pour etre interoperables avec d'autres systemes, peut etre que du JSON ou XML serait mieux ?

Si tu optes pour du CSV, ne le fait pas la main, utilise une bibliotheque pour generer ton fichier. En general c'est plutot l'operation inverse qui est complique mais d'experience creer un csv propre peut etre aussi casse gueule.
Je te conseil cet article : http://sebsauvage.net/wiki/doku.php?id=csv

Dernière modification par αjet (Le 31/03/2016, à 15:53)

---

αjet: ça se prononce alfajet, bordel ! | GMT+1 | Viens poueter avec moi, bordel ! | Mes photos | Shaarli | Fluidbuntu-fr

Feral

Re : Extraction de données => CSV [RESOLU]

Merci pour les info ajet.

Je n'ai aucune idée de quel language utiliser, je ne programme pas, je bricole juste un peu.
Je vais lire les liens que tu m'as donné et essayer de comprendre quoi/comment faire.

Merci!

credenhill

Re : Extraction de données => CSV [RESOLU]

hello

$ awk -F 'span class=\\\\u0022field-content\\\\u0022\\\\u003E|\\\\u003C|\\\\u003Cb\\\\u003E|\\\\u0022\\\\u003E' -v OFS=";" 'NF {gsub("\\\\", "", $44); print $8, $15, $24, $33, $44}' fichier
Nowra Veterinary Hospital;103 Kinghorne Street\nNOWRA NSW 2541\nAustralia;02 4421 5370;Dr Alison Havadjia, Dr Andrew Havadjia;www.nowravets.com.au
Claws and Paws Family Veterinary Hospital;44 Vittoria Street\nWEST BATHURST NSW 2795\nAustralia;02 6332 9096;Dr Caitlin Anderson;www.clawsandpaws.net.au
Gray Street Veterinary Surgery;Gray Street\nEMERALD QLD 4720\nAustralia;07 4982 4868;Dr Steven Bliss;
Gayndah Veterinary Surgery;2-4 Bridge Street\nGAYNDAH QLD 4625;07 4161 1404;Dr Nathan Hitchcock, Dr Sara Kahler;www.localvet.com.au/Gayndah/

Feral

Re : Extraction de données => CSV [RESOLU]

WOW, ça tue ça! ;-)
Merci Credenhill!

Je ne comprends pas grand chose mais c'est puissant.

Par contre je n'obtiens qu'une seule ligne sur une seule entité quand je l'applique à mon fichier. Saurais-tu pourquoi?

Mon fichier ressemble a ça exactement:

{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003ENowra Veterinary Hospital\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E103 Kinghorne Street\nNOWRA NSW 2541\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E02 4421 5370\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Alison Havadjia, Dr Andrew Havadjia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-website\u0022\u003E    \u003Cspan class=\u0022views-label views-label-website\u0022\u003EWebsite: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.nowravets.com.au\u0022\u003Ewww.nowravets.com.au\u003C\/a\u003E\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-34.8803","longitude":"150.602","title":"","markername":"","offset":"","":"\u003Cb\u003ENowra Veterinary Hospital\u003C\/b\u003E","opts":{"title":""}},{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EClaws and Paws Family Veterinary Hospital\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E44 Vittoria Street\nWEST BATHURST NSW 2795\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E02 6332 9096\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Caitlin Anderson\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-website\u0022\u003E    \u003Cspan class=\u0022views-label views-label-website\u0022\u003EWebsite: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E\u003Ca href=\u0022http:\/\/www.clawsandpaws.net.au\u0022\u003Ewww.clawsandpaws.net.au\u003C\/a\u003E\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-33.5418","longitude":"149.553","title":"","markername":"","offset":"","":"\u003Cb\u003EClaws and Paws Family Veterinary Hospital\u003C\/b\u003E","opts":{"title":""}},{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EGray Street Veterinary Surgery\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003EGray Street\nEMERALD QLD 4720\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E07 4982 4868\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Steven Bliss\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-23.5333","longitude":"148.15299999999999","title":"","markername":"","offset":"","":"\u003Cb\u003EGray Street Veterinary Surgery\u003C\/b\u003E","opts":{"title":""}},{"text":"\u003Cdiv class=\u0022gmap-popup\u0022\u003E  \n  \u003Cdiv class=\u0022views-field views-field-institute-name\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003E\u003Cb\u003EScarborough Veterinary Surgery\u003C\/b\u003E\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-postal-address\u0022\u003E        \u003Cspan class=\u0022field-content\u0022\u003ECorner Oxley Avenue and Silvester Street\nREDCLIFFE QLD 4020\nAustralia\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-work-phone\u0022\u003E    \u003Cspan class=\u0022views-label views-label-work-phone\u0022\u003EPhone: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003E07 3203 4095\u003C\/span\u003E  \u003C\/div\u003E  \n  \u003Cdiv class=\u0022views-field views-field-practice-info\u0022\u003E    \u003Cspan class=\u0022views-label views-label-practice-info\u0022\u003EAVA members at this practice: \u003C\/span\u003E    \u003Cspan class=\u0022field-content\u0022\u003EDr Candice McKeone-Taylor, Dr Maris Eastgate, Dr Tamara Dann\u003C\/span\u003E  \u003C\/div\u003E\u003C\/div\u003E","latitude":"-27.2187","longitude":"153.108","title":"","markername":"","offset":"","":"\u003Cb\u003EScarborough Veterinary Surgery\u003C\/b\u003E","opts":{"title":""}}

En mon résultat donne ça:

Nowra Veterinary Hospital;103 Kinghorne Street\nNOWRA NSW 2541\nAustralia;02 4421 5370;Dr Alison Havadjia, Dr Andrew Havadjia;www.nowravets.com.au

Merci beaucoup!

Dernière modification par Feral (Le 04/04/2016, à 06:43)

credenhill

Re : Extraction de données => CSV [RESOLU]

on definit -v RS='{"text"'  comme séparateur d'ennregistrement, à la place du saut de ligne

$ awk -F 'span class=\\\\u0022field-content\\\\u0022\\\\u003E|\\\\u003C|\\\\u003Cb\\\\u003E|\\\\u0022\\\\u003E' -v OFS=";" -v RS='{"text"' 'NF {gsub("\\\\", "", $44); print $8, $15, $24, $33, $44}' fichier
Nowra Veterinary Hospital;103 Kinghorne Street\nNOWRA NSW 2541\nAustralia;02 4421 5370;Dr Alison Havadjia, Dr Andrew Havadjia;www.nowravets.com.au
Claws and Paws Family Veterinary Hospital;44 Vittoria Street\nWEST BATHURST NSW 2795\nAustralia;02 6332 9096;Dr Caitlin Anderson;www.clawsandpaws.net.au
Gray Street Veterinary Surgery;Gray Street\nEMERALD QLD 4720\nAustralia;07 4982 4868;Dr Steven Bliss;
Scarborough Veterinary Surgery;Corner Oxley Avenue and Silvester Street\nREDCLIFFE QLD 4020\nAustralia;07 3203 4095;Dr Candice McKeone-Taylor, Dr Maris Eastgate, Dr Tamara Dann;

Feral

Re : Extraction de données => CSV [RESOLU]

Merci!
Ca marche a fond!
Impressionnant.