I had some expectations for the Nexenta session at Storage Field Day 6. They have recently made some changes in their management, modernized their website and marketing stuff (too much purestorage-like orange, IMHO) and I was hoping that things had changed for the better. But, after the meeting, I’m still very doubtful about their strategy and long term guidance.
A short introduction
I always liked Solaris and ZFS (I spent years of my professional life working with Unix and I was very fond of the Sun stuff) and when OpenSolaris was forked to Illumos I was worried about its future. The community is very small and the development process has begun to hobble somewhat.
Back in the days, a few companies in the valley secured Solaris/Dtrace/ZFS gurus that came out from the Sun diaspora. Just to give examples, Delphix has key figures from ZFS/Dtrace team (Adam Leventhal, Matthew Ahrens, Eric Schrock), One of the DSSD founders is Jeff Bonwick and Joyent’s CTO is Bryan Cantrill. These guys are leaders of the Illumos and ZFS communities, and most of the products of these companies have an Illumos DNA. It also happens that the heaviest Illumos contributors are Delphix and Joyent!
Who has Nexenta? None of them. I’m still wondering why a company like Nexenta, that is proclaiming itself as the leader in ZFS storage hasn’t manged to hire someone that makes the difference, someone who invented that stuff!
What really is the contribution of Nexenta to the community? They say a lot, but when you talk to people in the valley involved with Illumos and ZFS, Nexenta isn’t the first name that comes to their mind. Once again, if you check the OpenZFS summit agenda, most of the speakers are from Delphix!
At the end of the day, Nexenta has installations (in the order of 5500) and customers (half of the 5500 installations are enterprise editions, aka paying customers… but we don’t how much they pay!).
I’m not convinced about Nexenta story, but the new CEO (Tarkan Maner) is in charge only since the second half of 2013 and perhaps changing takes time.
NexentaStor: too many Software-defined constraints
I’ve tried, many times to use NexentaStor in real world implementations, but I’ve always failed. I usually work with enterprise end users (not handymen-like end users who have to spend too much time on making things work). Even though everyone likes Software-defined storage, my clients want to keep it simple.
My first problem comes from the HCL (hardware compatibility list): difficult to find, very limited and always outdated. It’s quite impossibile to build a fully functional, single vendor configuration with recent hardware. Every time I tried I had to build custom configurations with third party controllers, NICs, HBAs and disks. They don’t support PCIe flash, NVMe and many other new technologies out there. This is awkward and makes things very complicated from the support point of view. It’s true that you can buy certain pre-configured custom configurations from vendors like Dell, but it doesn’t help that much… reality is that you are forced to use very few hardware configurations and your freedom of choice is highly limited.
Problems aren’t only at the HCL level. The software sucks too. Active Directory integration is still problematic (especially with Windows Server 2012 and complex configurations), no SMB 3 support, SMB performance is still debatable in particular cases, deduplication is still unusable, hypervisor integration is limited, GUI is not the best you can find out there (I must say it’s improving though), Analytics (Dtrace) is not leveraged at its best, and the list goes on and on. There is always something missing to close the circle.
Others in the Open-ZFS storage space have the same identical problems but they managed to do it better. Cloudbyte and Syneto, for example, are proposing appliances but they keep software licenses separate from hardware and both of them have smarter UIs (very easy to use and well thought out for their user base), better hypervisor integrations. Cloud byte has solved the Illumos issue as well by proposing ZFS on top of FreeBSD.
Now that Nexenta has moved to Linux for its NexentaEdge (more on this later), what we can expect for NexentaStor?
Will they leave Illumos for Linux? It shouldn’t be that bad after all, Now OpenZFS is also available for Linux and it won’t be hard to port almost all Nexenta add-ons to it. Does Nexenta have a plan for that? Or why not?
NexentaConnect is pointless
We are talking about a NexentaStor VSA optimized to sit on top of VMware VSAN/ESXi to enable file protocols. It’s like having a VM with Microsoft Windows Storage Server but worse (in terms of protocols and AD integration) and more expensive, especially because most of the performance tweaks you can usually do on ZFS are not possibile in a VM (I’m thinking about ZIL, L2ARC for example). Most of the bloggers in the room (here the videos of the entire session) agreed that NexentaStor is more a Vmware partnering play then something really useful in the real world. You can find the same identical “Markitecture” design from Atlantis with its USX on top of VSAN here.
NexentaEdge: some choices are odd, but it’s too early to say if they are wrong
NexentaEdge is an Object Storage that has borrowed some ideas from ZFS and is developed on top of a standard Linux distribution. This is the third time I’ve been briefed about the product and I still have many doubts about some design choices. I’m not saying that they are good or bad, they are simply odd for an object store but it’s also too early to judge the results (product still not available in GA).
At the beginning the system will support S3 and Swift API as well as iSCSI. iSCSI is directly implemented on top of the cluster and it’s not very clear how much performance you can carve out of it or if NexentaEdge will be able to manage eventual or strong consistency…
But there is more,
– the extensive use of deduplication and multiple copies instead of simple erasure coding. When you talk about object storage, in many cases deduplication is ineffective because data is already compressed and/or encrypted at source. This results in a scarce deduplication ratio and, when this is added up to the number of copies you need to maintain data integrity, the overall efficiency is limited.
– Multicast and UDP. The product uses UDP instead of TCP for inter-node communication and leverages multicast to eliminate part of the chat between nodes involved in data movements. It looks very well implemented indeed, but it also strongly limits the number of possibile topologies (I’m not a networking specialist but, for example, implementing multicast in a multi-DC topology is very complex and risky).
At the end of the day, NexentaEdge looks much more like the backend of a general purpose multi-Petabyte scale-out storage than a classic distributed object storage. A product which, in time, could be able to serve many protocols and traditional workloads with a good performance. It would be a very interesting and ambitious move on Nexenta’s part (and it would change my mind about them). But, as I’ve already said, it’s too early to judge this product at the moment…
Intersting to note that NexentaEdge isn’t going to be open sourced… once again, it says a lot about the real Nexenta commitment to open source and Illumos… doesn’t it?
Final notes
Everthing Nexenta looks half backed, but it’s not bad per se. I don’t know the real strategy of the company and I can only make conjectures here:
1) Nexenta has a clear vision and strategy. At the beginning they took advantage of OpenSolaris/ZFS but then they realized that Illumos is a limping horse for them (especially because they didn’t invest when it was the right time and they don’t have good resources now). At the moment, they are developing a new product (NexentaEdge) leveraging their knowledge about ZFS and will compete in the nascent market of multi-Petabyte scale-out storage for the enterprise (something that I described here and where they will find some tough competition like Scality, DDN, Isilon, etc.). An immature product at the moment but with potential.
If this is true I’m sure that NexentaStor isn’t going to see a lot of improvements in the future… but who cares? Nexenta could manage a commercial upgrade to NexentaEdge when front-end features will be comparable (file protocols, replica, snapshots, UI, etc. ).
2) Nexenta has no clue of what it is doing. After years of struggling with a mediocre ZFS product, they are finally investing money (but not in NexentaStor), splitting the development team in two totally different entities and confusing the end user with half-backed products and bad communication about their commitment to open source. In this case I can’t see a great future for Nexenta.
I strongly hope, at least for Nexenta, that my first hypothesis is closer to reality than the latter. BTW, thanks for the nice orange shoes 😉
Disclaimer: I was invited to this event by the GestaltIT and they paid for travel and accommodation, I have not been compensated for my time and am not obliged to blog. Furthermore, the content is not reviewed, approved or published by any other person than the Juku team.
Glad you wrote this b/c I thought it was just me. It was even uncomfortable to watch from the stream. Quite confusing, and at the end, I really didn’t feel like I had a good handle on their direction.
I’ve been following what Nexenta does since the OpenSolaris ship started to sink in end 2009. Unfortunately your impressions and concerns covers to a high amount with mine, thanks for sharing them. Nexenta has so often changed its announcements, delayed products in the past and key people leaving for other gigs. The lack (also consistency in) communication isn’t really helpful to current and potential customers.
In the meantime other companies (as you mention) have stepped up. Some of them are iterating at a higher pace, not only as sealed products but some even to a large extent in the open (i.e. iXsystems with [True|Free]NAS).
Nexenta is a marketing company trying to become a real storage contender.
Stay away.
BTW, the same can be said about Tegile
My POV about Tegile is slightly different,
they based their OS on OpenSolaris/ZFS but they forked it and never joined the open source community. In fact, they don’t release their code or have never announced support to OpenSolaris/Illumos. They also modified ZFS in many aspects to fit their needs and I think they write their own drivers.
This strategy is really common indeed, most of the vendors out there use Linux or other open source OS at the base of their appliances without embracing open source at all.
Ouch! That was pretty hard to take, but I guess someone had to say it. I agree that NexentaStor was more a VMware play than anything else. And NexentaEdge makes no sense at this point in time, unless Nexenta has a whole lot of talent and the money it will take to make a splash in the object storage market. Most of the object-based storage software vendors (Caringo, Cloudian, Cleversafe, Scality) have been in business developing their software for years. The market for object storage is just starting to generate some momentum, and these companies are poised for some hard-earned success. Hard to imagine that a late entry like NexentaEdge will have the opportunity to get any traction in the object storage market, unless it is insanely great, which does not appear to be the case.