Can anyone revive AviationWX, PilotWX, or TAF

[sdetweil]

@mumblebaj but the code uses request…

It downloads a now gz folder, with one xml file in it… I tried to use fetch and couldn’t get it to unzip the file in stream…
2 days, gave up

I installed ok on a pi4 after doing my script install
warnings but no failures

npm WARN deprecated hoek@6.1.3: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated joi@13.7.0: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).

added 66 packages, and audited 67 packages in 46s

2 packages are looking for funding
  run `npm fund` for details

6 vulnerabilities (2 moderate, 4 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues possible (including breaking changes), run:
  npm audit fix --force

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
sam@raspberrypi4:~/MagicMirror/modules/MMM-aviationwx $ 

[sdetweil]

@FSAHD i just pushed a fix to package.json and a new postinstall script

git pull to get the updates…(watch out that it doesn’t overwrite your code)
then the postinstall script from the module folder

./postinstall

it will be called by npm install as well…

[mumblebaj]

@sdetweil I think it is possible with axios. Let me have a look at it. Will fork module from you and see what I can do and push a PR if it works.

[sdetweil]

@mumblebaj said in Can anyone revive AviationWX, PilotWX, or TAF:

I think it is possible with axios

really don’t want axios… better to get native fetch working…
thats what I was trying. and you can use node xxx.js to test it without MM in the way…

its ‘possible’. just all the different ‘it works like this’ didn’t work.

there are two fetches in the node helper… one gets the gz file
and one gets an xml file… that one should be easy, but I didn’t want to mix in the same code path

[mumblebaj]

@sdetweil The below code works fine. Tested it. Having issues with my system at the moment and can’t make the changes. Can you incorporate?

async function fetchData(metarUrl, airports) {
  try {
    const response = await axios.get(metarUrl, {
      responseType: 'arraybuffer', // ensure response is treated as binary
      headers: {
        'Accept-Encoding': 'gzip', // request gzip compression
      },
    });

    // Unzip the gzip response
    const unzippedData = zlib.gunzipSync(response.data).toString();

    let json_string = convert.toJson(unzippedData);
    var json = JSON.parse(json_string);
    var metarData = json.response.data.METAR;
    var airportData = new Object();

    var airports = payload.airportList.split(",").map(function (airport) {
        airport = airport.trim();
        return (airport.length < 4) ? "K" + airport : airport;
      });

    airports.forEach(function (airport) {
        metarData.forEach(function (metar) {
            if (airport === metar.station_id) {
                if (this.debug)
                    console.log("METAR data found for " + airport);
                airportData[airport] = metar;
                return; // check next airport in list
            }
        });
        console.log(metarData)
    });
  } catch (error) {
    console.error('Error fetching data:', error);
  }
}

[mumblebaj]

@sdetweil I can fix the other fetch as well no issues. Give me a few on this one.

[mumblebaj]

@sdetweil Both calls working.

try {
        const response = await axios.get(metarUrl, {
            responseType: 'arraybuffer', // ensure response is treated as binary
            headers: {
                'Accept-Encoding': 'gzip', // request gzip compression
            },
        });

        // Unzip the gzip response
        const unzippedData = zlib.gunzipSync(response.data).toString();

        let json_string = convert.toJson(unzippedData);
        var json = JSON.parse(json_string);
        var metarData = json.response.data.METAR;
        var airportData = new Object();

        var airports = payload.airportList.split(",").map(function (airport) {
            airport = airport.trim();
            return (airport.length < 4) ? "K" + airport : airport;
        });

        airports.forEach(function (airport) {
            metarData.forEach(function (metar) {
                if (airport === metar.station_id) {
                    if (this.debug)
                        console.log("METAR data found for " + airport);
                    airportData[airport] = metar;
                    return; // check next airport in list
                }
            });
            // console.log(metarData)
        });
    } catch (error) {
        console.error('Error fetching data:', error);
    }

    const checktype = { "Airspace_Flow_List":"Airspace_Flow", "Ground_Delay_List": "Ground_Delay", "Arrival_Departure_Delay_List": "Delay","Airport_Closure_List":"Airport" }

    
    try {
        const response = await axios.get(FAAUrl);

        // Your further processing
        let json_string =
            //convert.toJson(test_data)     swap comments to use test FAA data
            convert.toJson(response.data.toString());
        if (this.debug)
            console.log("json faa=" + json_string);
        const faa_data = JSON.parse(json_string);

        if (!Array.isArray(faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type)) {
            // save the current entry
            const x = faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type
            //  init as an array
            faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type = []
            // add the element into the array
            faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type.push(x)
        }
        // loop thru the delay types
        faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type.forEach(t => {
            //
            if (t.name !== "Airspace Flow Programs") {
                // for each the  there are two major keys, name and the list for that type
                // get the key of the list
                const keyname = Object.keys(t)[1]
                // remove the "_list"  part for passing to front end
                const key = keyname.split('_').slice(0, -1).join('_')
                if (this.debug)
                    console.log("key data=" + JSON.stringify(t[keyname][checktype[keyname]]) + "keyname=" + keyname + " key=" + checktype[keyname] + "\n")
                if (!Array.isArray(t[keyname][checktype[keyname]])) {
                    const x = t[keyname][checktype[keyname]]
                    t[keyname][checktype[keyname]] = []
                    t[keyname][checktype[keyname]].push(x)
                } else {
                    console.log("checkytype=" + checktype[keyname])
                }
                if (this.debug)
                    console.log("data=" + Object.keys(faa_data['AIRPORT_STATUS_INFORMATION'].Delay_type)[0] + " data=" + JSON.stringify(t) + " keyname=" + keyname + " key=" + key + " last part=" + JSON.stringify(t[keyname][key]) + "\n")

                // look thru all the aaffected airports for this record type
                t[keyname][checktype[keyname]].forEach(airport_record => {
                    // if the airport is one the user requested
                    if (airports.includes('K' + airport_record.ARPT)) {
                        if (airportData['K' + airport_record.ARPT]['FAA'] == undefined)
                            airportData['K' + airport_record.ARPT]['FAA'] = {}
                        // save this data record
                        airportData['K' + airport_record.ARPT]['FAA'][key] = airport_record
                    }
                })
            }

        });
        // Emitting data to Socket
        //self.sendSocketNotification("WX_RESULT", airportData);
    } catch (error) {
        console.error('Error fetching FAA data:', error);
    }

[sdetweil]

@mumblebaj thx. but there is a way to not have to use zlib. I just haven’t found any that work

[mumblebaj]

@sdetweil If you want to use fetch you can do the first call as follows.

try {
    const response = await fetch(metarUrl, {
      headers: {
        'Accept-Encoding': 'gzip', // Request gzip compression
      },
    });

    if (!response.ok) {
      throw new Error(`HTTP error! Status: ${response.status}`);
    }

    // Read response body as buffer
    const buffer = await response.buffer();

    // Manually decompress gzip data
    const zlib = require('zlib');
    const decompressedData = zlib.gunzipSync(buffer).toString();

You could use pako.

const decompressedData = pako.inflate(buffer, { to: 'string' });

[sdetweil]

@mumblebaj but… you are not supposed to have to do either