This appeared on Facebook regarding the IT situation over at the ARRL:

This was reported by Chris NW6V 

 Mike Ritz, W7VO, ARRL 2nd Vice President, was at the WVDXC club meeting last night, having just attended an ARRL board meeting to discuss the outage. Mike reported those meetings are a weekly event until the matter is resolved.

Everything that was running on INTERNAL servers is down until further notice. That includes their VoIP phone system, their .org email addresses, and front ends for things like LOTW. Everything running on external servers - cloud servers etc. - including LOTW data, is believed unaffected. But, such data will not be available until the internal matters are resolved. Thus, "joe@arrl.org" doesn't work - because that was on their internal mail server. But "joe@arrl.net" does - because the relays didn't run internally. I checked, and NW6V@ARRL.NETworks.

Efforts to restore the internal systems is proceeding full-time. No time-line can be given. The nature of the problem cannot be discussed.

I believe Mike said - with air asterisks around his words - "We have been advised to say nothing." He responded similarly when asked if "the Feds" were investigating this.

Being that the ARRL is connected to Homeland Security through its disaster response functions, and that personal data (no credit cards) for many relatively important persons are stored in the systems (business, military, science, etc.) such an investigation could very well involve the FBI and Homeland Security.

I (Chris NW6V)  was an IT Director in mental healthcare for many years, so security was a big part of my responsibility. As a professional looking at it from the outside, this has all the earmarks of a hack - of sufficient severity that it needed to be reported as a CRIME. At which point, IT is required to lock everything down - every computer and device involved becomes EVIDENCE - until a full investigation by forensics experts - da cops - is conducted. Getting everything back up is NOT job 1. Once the "crime scene" is clear (yellow tape down), THEN the job of recovery can begin. If some kind of hack had wormed into the ARRL system, recovery of local system by restoring backups becomes problematic - it may be difficult to verify that backups contain no trace of the hack. In which case, recovery and restoration of services would be slow and very painful.

This fits what we know about the situation to a "T."

73 Chris NW6V

Interesting. If the ARRL was hacked by someone with some kind of revenge motive in mind, they may have bitten off way more than they could chew. I never thought of the possibility of Federal Government involvement in this due to ties and MOEs with Homeland Security.  If this was a hack, and it is determined to be a crime, the perpetrators could be facing some very serious Federal prison time if caught and prosecuted.

I hope this gets cleared up soon. The LOTW situation doesn't bother me so much, but any disruption to the VE process does. My Team's next session is a week from this coming Saturday - June 8th. I'd hate to have to tell my candidates that I have no idea as to when their results will be processed. And no sooner did I write this, than I discovered this bit of good news:

ARRL VEC Services Update During Systems Disruption

05/29/2024

ARRL previously reported that we are responding to a serious incident involving access to our network and headquarters-based systems. Several services have been affected, including those administered by the ARRL Volunteer Examiner Coordinator (ARRL VEC).

Exam Registrations and Materials. ARRL Volunteer Examiners (VEs) should continue to submit exam registrations and material requests. While we are unable to post new or revised exam session dates and details to the website, we can ship out exam materials. Please remember that most exam materials are available on our website (www.arrl.org/resources-for-ves).

Processing Applications to the FCC. We have resumed the processing of Amateur Radio License applications with the FCC. This includes applications for new and upgrade licenses, individual applications, and club license applications. Exam sessions will be submitted to the FCC in chronological order, from earliest test dates to the latest. Please allow additional time for our processing as the exam session backlog is cleared.

International Radio Permit and License Class Certificates. We are currently unable to create International Amateur Radio Permits, License Class Certificates, and Volunteer Examiner (VE) badges, certificates, and stickers. New ARRL VE applications and renewals are unable to be processed at this time.

ARRL Youth Licensing Grant Program | FCC Application Fee Reimbursement Information. ARRL is continuing to accept reimbursement forms to cover the one-time $35 application fee for new license candidates younger than 18-years of age for tests administered under the auspices of the ARRL VEC. Reimbursement checks may take longer than normal to be processed at this time.

ARRL VEC cannot be reached by telephone at this time. We appreciate your patience as ARRL continues to work on restoring access to affected systems and services.

This story will be updated with new developments.

Thank you ARRL VEC Department for the update!

72 de Larry W2LJ

QRP - When you care to send the very least!

Good news! Remote VE Testing is happening and people are getting licensed and upgraded! Woohoo!

I spoke with Brandin Hess, AL6I, Anchorage ARC VEC chairman on the phone shortly after this test session was complete. He told me that remote testing is well underway. At the current rate, they can handle between 5 and 10 candidates a day, possibly up to 20 if everyone was operating at full tilt. Only 40 applications are in process right now (and many more are sure to come). Several applications have been rejected, in part due to the an in-person proctor not being available because of shelter-in-place requirements.

I want to publicly thank Brandin and the Anchorage VEC for their efforts. He says it’s using up 110% of his time, and taking an impact on his health, but if he wasn’t working on it, ham radio licensing would come to a screeching halt.

The process at Anchorage still requires an in-person proctor, who is essentially a trusted individual, to verify that the test taker is following all the rules and not cheating, and to – without a shadow of a doubt – validate the integrity of the test session per Part 97 regulations.

But what if it’s okay to perform testing entirely online? A working group is currently underway solving that problem.

In the last few hours a group of about 30 people has coalesced from across the country, all working towards defining specific goals and requirements for 100% online amateur radio examinations, performing proofs of concept, and creating a standard operating procedure that any VEC can adopt and modify for their needs. No software is being written, yet, as it may very well be possible this can be done with off-the-shelf components like video conferencing, Docusign, ExamTools, and even mirrors!

The effort got quite a massive signal boost with the help of Sam Hulick, renowned composer and software developer, and verified “blue checkmark” Twitter user, sent a tweet to Ajit Pai, FCC Chairman. Yes. The Ajit Pai. And he responded:

I eagerly await to see the results of this tweet, but at the end of the day, there’s no need for any FCC action, besides maybe a and a vote of confidence that VECs can reliably perform 100% remote testing.

This clearly isn’t an effort that will result in immediate adoption (although theoretically the capability is already available), but it will – within the next few months or less – result in a framework and/or set of tools that VECs and their VE teams can use to perform remote testing reliably and be confident that it is FCC compliant.

But is it really FCC compliant? By that I mean, does a 100% online synchronous exam with several angles of surveillance of the test taker by 3 or more VEs meet all subparts of Part 97.509? I think so. What do you think?

I will keep the updates flowing and those interested informed as progress is made. If you want to be added to the conversation, reach out to me.