{"id":187,"date":"2012-06-08T22:13:21","date_gmt":"2012-06-08T21:13:21","guid":{"rendered":"http:\/\/www.hilltop-cottage.info\/blogs\/adam\/?p=187"},"modified":"2012-06-08T22:17:31","modified_gmt":"2012-06-08T21:17:31","slug":"bulk-upload-of-historical-weather-data-to-wundergound","status":"publish","type":"post","link":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/bulk-upload-of-historical-weather-data-to-wundergound\/","title":{"rendered":"Bulk Upload of Historical Weather Data to Wundergound"},"content":{"rendered":"<p>A few months ago I bought a weather station but only lately registered with Wunderground and was slightly dismayed that the software I use (<a href=\"http:\/\/sandaysoft.com\/products\/cumulus\" target=\"_blank\">Cumulus<\/a>, which is heaps better than what the station supplier included) does not post historical data; it only uploads to Wunderground what it downloads from the station when Cumulus starts up.<\/p>\n<p>It seems there is no &#8220;off the shelf&#8221; way of doing this so here is a how-to-do it. The steps should be applicable to any weather station software that stores the data as a &#8220;CSV&#8221; or formatted text file, with a little bit of fiddling. I guess a similar process works for weather upload sites other than Wunderground too. The steps are:<\/p>\n<ul>\n<li>get the stored data into a spreadsheet. This is easy for Cumulus since the data is stored in a text file with commas separating the values and the Cumulus help file tells you what each column is.<\/li>\n<li>convert the units. I prefer SI units but Wunderground uses American conventions.<\/li>\n<li>create a URL (web address), one for each data record to be uploaded. The structure for these is <a href=\"http:\/\/wiki.wunderground.com\/index.php\/PWS_-_Upload_Protocol\" target=\"_blank\">documented on the wunderground wiki<\/a>. You can simply copy these into a web browser (Firefox, Internet Explorer etc) one by one but once you&#8217;ve had the satisfaction of seeing it work, the remaining thousands of records need a better method.<\/li>\n<li>use wget to process a batch of URLs. Wget is free open source software. Windows users can <a href=\"http:\/\/gnuwin32.sourceforge.net\/packages\/wget.htm\" target=\"_blank\">find in installer here<\/a>.<\/li>\n<\/ul>\n<p>I&#8217;ve uploaded an <a href=\"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-content\/\/2012\/06\/Example-Spreadsheet.xls\">example spreadsheet<\/a> that does the conversion and creates the URLs. Notice it has three sheets and that the &#8220;Conversion Factors&#8221; also contains the base of the URL. This must be edited to have your station name and password. NB there is also a Celcius-Farenheit conversion and the peculiar column L on the &#8220;Converted&#8221; sheet is needed to get a correctly formatted URL from the date-time combination.<\/p>\n<p>To use wget I copied the URLs into a &#8220;.txt&#8221; file, one URL per line (just drag down the entries in the column of the spreadsheet and copy\/paste into an empty text file) and created a small &#8220;.bat&#8221; file containing:<\/p>\n<p><strong>PATH &#8220;C:\\Program Files\\GnuWin32\\bin&#8221;<\/strong><br \/>\n<strong>wget -v -o log.txt -O responses.txt -i &#8220;urls Apr12.txt&#8221;<\/strong><\/p>\n<p>You may need to alter the first line depending on where you installed wget. The second line assumes the list of URLs was saved to &#8220;urls Apr12.txt&#8221;. The &#8220;log.txt&#8221; file contains verbose logging of each &#8220;wget&#8221; and the &#8220;responses.txt&#8221; file should just contain a long string composed of one &#8220;sucess&#8221; for each sucessful posting of data. If it doesn&#8217;t, something went wrong&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A few months ago I bought a weather station but only lately registered with Wunderground and was slightly dismayed that&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-187","post","type-post","status-publish","format-standard","hentry","category-uncategorized","post-archive"],"_links":{"self":[{"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/posts\/187","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/comments?post=187"}],"version-history":[{"count":4,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/posts\/187\/revisions"}],"predecessor-version":[{"id":190,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/posts\/187\/revisions\/190"}],"wp:attachment":[{"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/media?parent=187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/categories?post=187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hilltop-cottage.info\/blogs\/adam\/wp-json\/wp\/v2\/tags?post=187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}