Open source software development

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Parallel to a little open source drama this morning...

As a longtime proponent of open source software, the fact of the matter is that we're still surrounded by proprietary software, and it isn't trivial to get away from that. I'm perhaps dating myself a little bit here, but I got to thinking about this a bit and found it to be fascinatingly deep. Years ago, we built many systems out of discrete logic, because CPU's were expensive, complicated designs. You used discrete circuits to encode and decode, do logical/arithmetic operations, etc., and indeed many early computers were built out of discrete logic components.

In the '80's, a conventional personal computer might consist of a 6502 CPU (4528 transistors), a few specialized I/O chips for audio, video, and other I/O, some discrete memory chips for storage, and then other miscellaneous logic to tie it all together. It was probably possible for a human to fully grasp what was going on inside these systems, and the inclusion of circuit schematics in things such as Commodore's C-64 Programmers Reference Guide certainly encouraged that.

At some point, probably in the '90's, electronic design had significantly shifted, and we started to see heavy use of microcontrollers in many applications where discrete logic had formerly been the solution. This makes a certain kind of sense, because a microcontroller can be reprogrammed without redesigning a circuit, and is capable of very complex logic in a very small package. As of 2019, certain microcontrollers are as small as a grain of rice (ATtiny) - and dirt cheap. This has translated into these things being everywhere, being used for everything.

As our computers have become more complicated, though, we're seeing more and more subsystems built in. Most of these have firmware or even complete operating systems, and many are not open source, some can't even be easily investigated.

Think of all the devices directly attached to your computer which are powered by software/firmware.

Start at the keyboard and mouse, both of which have firmware or even upgradeable software. Moving towards the computer, the typical USB controller is a software-powered device. I/O controllers are frequently powered by firmware, and devices such as an HBA are actually full CPU's with a lot of access into your system. I/O devices such as hard drives, SSD's, printers, etc., also have their own CPU and firmware. The server/PC CPU/chipset itself these days has things like the IME, with full platform access, and that's sometimes supplemented by IPMI/BMC. Graphics subsystems are their own compute powerhouses, and it's easy to forget that even your monitor is probably a tiny computer driving a standard LCD display. Your nice wireless Bluetooth headphones -again, a microcontroller in there. Your power supply might be reporting to your motherboard using a microcontroller. Even the FANS on your server are likely to have a little microcontroller in them to convert the PWM and to drive the tachy signal. And anyone who wants to point out that I missed any of a hundred other embedded firmware-driven devices along the way -- YES YOU ARE CORRECT! Feel free to list them below if you want! This is just my super-incomplete list of obvious ones.

While I appreciate the desire for everything to be open source software, the reality is that it is very difficult to assemble an environment where that is actually so. I don't know that there's a realistic way to get to the all-open-source model.

To that end, I'm not really sure why the hot issue of the last 24 hours is "Jira isn't open source". Surely it is more of a realistic operational concern that LSI HBA firmware isn't open source, or that the system BIOS code isn't open source. As much as I'd love for there to be a free open source competitor to Jira that was feature-for-feature comparable, the truth is that open source software often lags in development funding, because people download it and use it for free, leaving the developers to struggle to come up with a strategy (such as paid support) to fund development.

I, for one, very much appreciate that iXsystems has released the vast majority of TrueNAS as FreeNAS and made this freely available. iXsystems has done massive amounts of work to improve the state of ZFS on FreeBSD, and (IIRC) has sponsored stuff such as @mav@'s kernel iSCSI work. I've had accounts on Jira, Redmine, and other bug management systems, and they all have their ins and outs, but Jira *is* really nice. Compare and contrast to people forced to work under tragic systems such as debbugs ... such developer frustration when tools that are literally a quarter of a century newer are available.

Where the choice is an equal or nearly-equal one between an open source and a proprietary solution, I try to favor the open source one. And I try to contribute back somehow. Which is why I'm here, posting as many helpful replies as I can reasonably afford the time to, because I do appreciate FreeNAS, ZFS, and iXsystems. But if the developers want to use Jira to manage their bugs because it is so much better than the alternatives, I am definitely fine with that.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
To that end, I'm not really sure why the hot issue of the last 24 hours is "Jira isn't open source". Surely it is more of a realistic operational concern that LSI HBA firmware isn't open source, or that the system BIOS code isn't open source.
This! Who cares about the bug tracker being proprietary? It's a tool that barely interacts with the product and if it fails, the devs are sad but little more happens.

Firmware, on the other hand, is nearly always proprietary and extremely buggy. That is where outrage is merited. Take an HBA. As a PCIe device, it can do DMA. It's running a binary blob on a fairly beefy processor, by embedded standards. Who's to say the HBA isn't compromising my NIC and slowly piping my data to China?
The HBA could conceivably do that, a bug tracker not really.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Who's to say the HBA isn't compromising my NIC and slowly piping my data to China?

I like it! That’s properly paranoid. That’d mean you must encrypt all your datastores. People have had trouble with that, though, on account of “keeping track of keys is not intuitive” and “if it gets 100% full you may not have enough space to decrypt”.

The HBA is lying in wait. Waiting for that specific juicy piece of data / phase of the moon to communicate, lest it be found out by your border firewall (“why does my monthly report show outgoing traffic to China? From my FreeNAS boxen?”). Be afraid. Be very afraid!
 

Tigersharke

BOfH in User's clothing
Administrator
Moderator
Joined
May 18, 2016
Messages
890
Obviously in answer to a thread elsewhere, and while typing this comment an obscenity was displayed in the originating thread with likely significant consequences for the poster, and the thread was closed I believe.
FOSS can be religion, but let’s not, eh? Next we’ll mandate that FOSS developers can’t code on Windows or Mac machines, has to be FreeBSD or a Linux distro without closed-source drivers.

Leave religion to the pope.
That was exactly my thought, although it was worded more as "wow how Stallman-like" which, is a big reason I prefer BSD since it is much less a viewpoint of prescription and I feel fits better with the Unix concept that one is free to do even dangerous things (ie, root could rm -rf from any path, and nothing, or very little, be excluded). I was unaware of the 'free license' to Open Source projects which may be fairly common, but so long as I am not adversely affected I don't even give it a thought. I also believe there is far more silently, unattributed, unacknowledged, donations back to Open Source than we will ever know. It need not ONLY be code or money, but time and knowledge. When a community is welcoming, and would not trample upon the person who might provide insights because their day job is at Homeland Security or NSA or some other place that cannot risk involvement, the community aka Open Source wins, gains. I don't really care to be told what I must do, and many in the BSD world agree, they do what is best because they believe in what is best, and they give back because that is who they are. A simpler license also takes effort and headaches away when only a few easy points need be considered. There have been other licenses since people have been sharing software, before the Internet got big and utility-like, and well before Stallman, Linus, got involved too much, and before GPL was seriously considered. It is easy to investigate the multitude of licenses that have ever been used, but I can nearly guarantee you that the majority expect attribution for the code, because even if the programmer is altruistic in every other sense, they want some tiny bit of recognition for their effort.

We should all be happy that a company can contribute by way of BSD-licensed software which is free, but there is no requirement that it must be so.

Also, we should be glad when coders 'dog food' or we would be in worse shape and likely have far many more issues to track or bugs to report.

EDIT:

I certainly would not be doing theme work for any commercial-only site. They most likely would not have given me all the access to do any of the things I do here. In a commercial environment (say Microsoft Forums for example), I bet they would need to hire me or at the very least require a signed NDA to even contemplate CSS work for their forum. Everywhere that I cannot contribute such CSS enhancements directly, I use stylus on my browser to make things nicer for myself. I had once uploaded some CSS to the stylish theme share, and could do the same for various CSS I assemble, I have to find the best way/place though.
 
Last edited:

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Interesting point on licensing. Linus has stated publicly that they prefer GPLv2 over GPLv3 precisely because it allows someone to take the kernel, improve upon it, use it internally for great gain, and not share that code back. Biggest example of that is Google.

In Linus’ view, anything that encourages corporations to use a FOSS kernel over a closed source one is good. Some contributions will come from it, even if other pieces remain proprietary.

GPLv3 would mandate sharing the source back even when the resulting kernel is never distributed and only used internal to a company.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,449
Firmware, on the other hand, is nearly always proprietary and extremely buggy. That is where outrage is merited. Take an HBA. As a PCIe device, it can do DMA. It's running a binary blob on a fairly beefy processor, by embedded standards. Who's to say the HBA isn't compromising my NIC and slowly piping my data to China?
The HBA could conceivably do that, a bug tracker not really.
Firmware is always buggy because a lot of people working with microcontrollers/CPU don't necessarily understand what is happening at the hardware level.
Most often, there is the abstraction layer the programmer doesn't want to deal with and as a result focus on high level or project oriented programming.
Software people don't necessarely understand concurrent processing and time relationship.
Often, because the code doesn't return errors, doesn't means the intended feature is working as it should.
If you design FPGA's, you can work at the cell level or do higher level programming.
If DMA is proprietary, it is in its implementation. Nothing too difficult designing one when you know how.
With FPGA's it is not enough having the correct implementation, it has to be verified at every steps of the way becuase it is just too critical and easy to make the simpliest mistakes
FPGA's are more complexe than CPU's, because CPU functions have been already validated.

The reason software is so buggy, is mostly because there is no real and consistent use of code and function validation, such as Lint, code coverage...

@jgreco, the reason open-source is so difficult, is not just because of fundings. As far as understand, with open-source mentality, any one can contribute, but the issue also is to coordinate everybody which open-source can't do much about.
As a result, you will find one project being branched to a different project.
What started as a good idea, now become scarce.
Having and entity with an open-minded open-source concept such as iXSystem, Nextcloud can succeed, because they can benefit from the financing they get from open-source development and backed by the open-minded users.

With closed-source programming and such, the managing entity must focus on project development and to do so has clear agenda on what has to be done, hence not spreading their team thin.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
and we started to see heavy use of microcontrollers in many applications where discrete logic had formerly been the solution.
Orthogonal to the point of your post, but wide and cheap availability of microcontrollers (and the IDEs to develop for them, e.g. Arduino) has turned a lot of things that used to be hardware problems, into software problems. I'm not a hardware guy (or even particularly a software guy), but someone asked me a few years ago to come up with a pressure-activated blinkenlight that turned on below a certain pressure. My design was a Teensy, an InAmp, and various passives to make it all go, with the pressure sensing and blinking done in software. Overkill? Definitely. But simpler for my mind to deal with.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,449
Orthogonal to the point of your post, but wide and cheap availability of microcontrollers (and the IDEs to develop for them, e.g. Arduino) has turned a lot of things that used to be hardware problems, into software problems. I'm not a hardware guy (or even particularly a software guy), but someone asked me a few years ago to come up with a pressure-activated blinkenlight that turned on below a certain pressure. My design was a Teensy, an InAmp, and various passives to make it all go, with the pressure sensing and blinking done in software. Overkill? Definitely. But simpler for my mind to deal with.
Overkill, not so much I think.
Nowadays, I find to be more challenging to keep to discreet components, due to the high level of integration.
Almost everything can now be done in the digital domain.
And to do that you need firmware and software.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Interesting point on licensing. Linus has stated publicly that they prefer GPLv2 over GPLv3 precisely because it allows someone to take the kernel, improve upon it, use it internally for great gain, and not share that code back. Biggest example of that is Google.

In Linus’ view, anything that encourages corporations to use a FOSS kernel over a closed source one is good. Some contributions will come from it, even if other pieces remain proprietary.

GPLv3 would mandate sharing the source back even when the resulting kernel is never distributed and only used internal to a company.

Yeah, well, GPLv3 is not a free software license for reasonable values of "free." It might succeed at keeping the SOFTWARE "free," but in placing significant restrictions on how you can use it, that stops it from being "free."

The old saying about the BSD style license applies. There's copyright (the usual), copyleft (GNU), and copycenter (BSD - take it down to the copy center and make as many copies as you want, doing what you want/need/etc).

I arranged a bunch of funding for Diablo over the years by using a contract that specified that the work product could not be considered "work made for hire" and that they would instead be given an exclusivity clause for a limited period of time, six months to a year, depending. That brought a lot of nice features to see the light of day and contributed quite a bit to the growth of USENET at the major players. There are lots of ways to do things that allow companies to benefit without necessarily keeping the code proprietary...
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
(not that anyone cares, but this is the project that had someone asking me to come up with the blinkenlight). But on-topic, the software (such as it is) is open-source.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,449
(not that anyone cares, but this is the project that had someone asking me to come up with the blinkenlight). But on-topic, the software (such as it is) is open-source.
I had a quick look at your project and I noticed a few things wrong about it.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
a few things wrong
I'd be interested in hearing about them, though it's no longer under active development--it does what I need it to do, even if setting the time is much more of a pain than it ought to be.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Ah, descrete components, I can easily relate. When a PC first came to market there was very little hardware and software was terrible and only a "computer geek" could love it. It was up to the individual like myself to create new circuits and software to make the computer more useful. Good times! Whatever I developed was always available to anyone who wanted it, but these were pre-internet days, we did have Bulletin Board Systems (BBS). For me the hardware design was the most enjoyable, the software design was enjoyable too because it allowed me to see my project actually work as intended.

Now to the OP's posting: I'd love to see an open source world but unfortunately our world isn't there yet and may never be. I too like open source projects such as FreeNAS and to contribute to them. I did contribute much more in it's early days but my life has changed and doesn't afford me as much time to play with FreeNAS as I use to. I think it's great that there are lots of open source program out there but there are products that are proprietary that may be more polished or just have a desired feature set that someone is willing to pay for and it makes sense to them over using a like open source product. And for many companies this decision could be a IA concern, not saying open source can't pass IA but if a product that is proprietary already has the credentials then it may get the green flag because it's a more cost effective solution or timely solution.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
but these were pre-internet days, we did have Bulletin Board Systems (BBS)
I remember those--I used to run one, in fact. To the best of my knowledge, I was the first one in my area with Internet email (along with UUCP-like newsgroup feed) in the mid-'90s. Long-distance bills were pretty bad, though.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I was the first one in my area with Internet email (along with UUCP-like newsgroup feed) in the mid-'90s.

Mid-90's? Latecomer. ;-)

It was all very much nicer before Eternal September.

Was operator of two of the region's three large UUCP hubs here.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Now can you imagine what the kids of today will say in 30-40 years? If they have brain implants that directly feed data into your brain and make every situation feel like you are really there and the huge amounts of data you could assess in an instant, they will be talking about the good old days when you had to use a computer monitor and a keyboard for input and output, and how Gigabit Internet use to be considered fast, not a neural interface. Of course there still will not be a cure for the common cold. ;)
 

Tigersharke

BOfH in User's clothing
Administrator
Moderator
Joined
May 18, 2016
Messages
890
There are a number of reasons why I do not own/use a 'smart' phone, the primary being control. A smartphone is a hardware blob that runs multiple binary blobs. The user is greatly isolated and has only the smallest degree of control that the manufacturer or app developer grants. For me, this is entirely contrary to the concept of FOSS, its promise always having been access to the code which also implies control.

@jgreco What is/was 'Eternal September'?

The internet was "becoming a thing" in around 1991, even GNOME had its logo discussion around then. I would guess that the explosion of commercialism on the internet was happening near 2000, up to then they still mailed out those shiny AOL coasters. By 1994 or so I had largely left BBSes behind in favor of a shell account from a local isp. We all hoped and wished, back in those imperfect modem days, that fiber-optic service would be rolled out, but we were "gifted" broadband instead. Now they are trying to get fiber-optic rolling *finally* but they may as well be crawling up Everest bare kneed.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
Sometimes I feel bloody old!

My first international E-Mail, instant messaging, group chat and news group reading / posting was before 1980. And yes, I really mean late '70s. It may not have been the Internet we have today, (which I did get dial up mid to late '80s), but it was better than the early days of the Internet. If you don't believe, (I would not, unless I knew), look up Plato (computer_system) on Wikipedia.

Of course, being an Elf, I live longer and look younger than you humans. (My hair is still almost all brown... :smile:.
 
Top