server uploading specific pages to clients

Is kJams missing a feature you need? Post it here! Note: if iTunes has it but kJams doesn't, it's a good bet kJams will have it, but that I just haven't gotten around to it yet. But go ahead and request it anyway!

How many people think this is awesome?

Poll ended at Wed May 06, 2015 12:50 pm

yeah i'd be stoked for this
2
67%
other things are more important
1
33%
 
Total votes: 3

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

server uploading specific pages to clients

Post by DeusExMachina »

Would it be possible to allow a client-originated server request to be formatted such that the server interpreted it as a request for a certain page in the DB.
This would work similar to adding a tag to the current search, such as "artist:" to restrict the search to that field.
In this case, you might ask for the first n entries of a particular place in the db, either by position or by alphabetical order. So, e.g., a request might look like the following:
p:"b":25
or
p:.5:30
Here a value in quotes is interpreted as the first entry starting with that particular character string, and the entire call would be the first Page of listings that start at the letter "b", returning 25 entries per page.
In the second example a non-quoted numeric signifies an approximate percentage position (50% in the example) into the library.

What this would then allow is for book-style browsing, as we discussed earlier, to be implemented directly in the current web client architecture. A tabbed alphanumeric listing could allow people to browse alphabetically, tabbing into the song book at letter boundaries and then page forward and back from there. Furthermore, a slider could allow the user to flip immediately to a given position in the library, say a third of the way in, and page forward and back from there.

As far as I can see it, the only issue would be implementing a way to restrict the search to NOT searching for terms inside terms, so "B" would only match words that began with "B", not words that simply contained it. While this may be nontrivial in the current architecture, the benefits would be huge for reasons already discussed in previous communications. That said, I suspect it WILL be trivial in the fast search SQL db.
This architecture, if implemented now, could also be utilized in any new mobile app under development.
Last edited by DeusExMachina on Sun Mar 08, 2015 1:48 pm, edited 1 time in total.

dave
Site Admin
Posts: 6684
Joined: Sun Sep 18, 2005 8:02 am
Location: Seattle
Contact:

Re: server uploading specific pages to clients

Post by dave »

an interesting idea. i think we've toyed with this idea in the past. not too difficult to do the "position" based list, a bit more work for the "alpha" based list.

we already cache the sorted list of songs (separately from what may be showing in the Library), so it would be a simple thing to say "okay give me this list again, starting from position N give me M entries. you could page forward and back from there, with the browser being responsible for keeping track of N as the user pages.

for "alpha" based it would have to be a bit more complicated: we'd say "give me that sorted list, starting from letter(s) 'xyz', offset by N, and give me M entries", with 'xyz' being the string to START the search with, N being a positive or negative offset (usually the same as M) and M being the number of entries you want.

it's possible to do this in the current design, and it will "just work" when i do eventually move to the new design (this feature is written at a higher level than direct DB access).

however, i'd have to see a lot of votes for this feature, to get it high enough on my list to consider it soon.

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

Yes, we toyed around with it before. In fact, IIRC we already implemented one aspect, the ability of kJams to serve progressive pages on client request.
The idea is to get a browsable songbook using the currently existing architecture (as opposed to having to craft an entirely new one) that is flexible enough that others can use it (e.g. Sargent) to create their own interfaces.
(Note: while the browser can keep track of N, either as a session variable, or in a cookie, I would have thought it easier on the server side, via a session variable stored along with the other session variables that let kJams know who is who and what session belongs to what client. But you know far better than me! :) )

Re: the alpha-based list, I actually had something a bit simpler in mind. On clicking a book icon, they would be presented with a scrub bar and alphanumeric tabs. Navigation would either be random access, via the scrub bar (with a page number box) or to specific sections based on alphanumeric position, by clicking one of 27 tabs (one for each letter plus one for other chars). (Not sure how this internationalizes to different language scripts.) So no need for getting the sorted list starting at "xyz", only starting at "x". From there the user would just page forward and back, possibly by multiple pages.
(I suppose it would be nice if clicking on a letter tab brought up a second row of tabs (e.g "B" -> "Ba", "Bb", "Bc", etc.) but this seems unnecessary, and, as this example indicates, a bit wasteful in terms of UI real estate, since many tabs for many indices will have no entries.)
Regardless, from this point, the client would just request another page, the next page or the previous page, something we discussed previously, and which I think you told me you had already written.
So the system would really only have 27 key entry points, and would page forward and back from there, with n entries per page. Alternately, once they were at a given section of the alphabet, they could jump farther distances with the scrub bar/page entry field.
All that said, couldn't kJams just keep track of the letter boundaries when it did the initial sort, the item # where each new letter section starts (LetterIndex="A"; if currInitialLetter<>LetterIndex then BIndex=CurrEntryNum and LetterIndex="B", etc..
Then when a client asks for the first "S" page, via p:"S":25, it just returns the first 25 items after the item number it has stored that corresponds to the first "S" entry.
This would seem to accomplish the same thing, with no need to deploy any search at all.

Regardless, all that said, if that is too much work for the present time, how about this:
since the "bit more complicated part" stems from the need to sort based on word starting chars (assuming my above alternate algorithm won't work for some reason) as a first step, how hard would it be to provide the ability to do just the random access (scrub bar) search? So, a search query crafted as p:.5:25 (the decimal is important) would send to the client a "search result" that consisted of the first 25 entires at the halfway point in the sorted list. It would also return a "page number" calculated by kJams based on the existing sorted list and the specified page size, possibly as the first (or last) item in the returned list. This page number could then be displayed in a text box, along with page forward and back elements. The user could then page forward and backward from there, which would add or subtract x from n, and send that page number as a request to kJams, via a request formatted like this: p:PageNum:ItemsPerPage.
E.g., p:1125:25
This would return items 28,125 (1125*25) through 28,150 in the sorted list as a standard search result.
Note: reals would be interpreted as percentages, and integers would be interpreted as page numbers, simplifying the formatting of the request string.

So, for instance, Emily sits down at a machine with no idea what she wants to sing (a common occurrence) so she can't really search for anything. So she goes into book mode, clicks in the scrub bar about where she thinks the "S" section might be (just at random, like opening a song book 3/4 of the way through) which is sent to the server as a search of the form p:.75:50 and is presented with a page of 50 entries 3/4 of the way into the list. In addition, the page number "1,009" is displayed in the page number field.
This is in the "R" section, so she clicks the ">" button a couple times, which gets sent to kJams as a "p:1010:50" and "p:1011:50" search request. Since this is not getting her to the S section fast enough (she is looking at a page currently listing "Rogue") she then types "1050" into the page number field, and is shown a page of 50 entries starting at "Simon, Paul & Art Garfunkle Mrs. Robinson".
One additional item would be how the internal list is sorted. Is it only sorted in the current sort order? If so, and that sort order is a non-useful sort order (PID say) then kJams would have to sort on Artist or Name first, since it is getting loaded from a caged sorted list. IIRC, multiple sorts are cached, and if this is true, then as long as these two are stored, we're good, and save the time needed to re-sort.
That said, I guess we have to modify the query format a bit, something like:

p:PageNum:ItemsPerPage:a (for artist sort) and p:PageNum:ItemsPerPage:n (for song name sort).

Alternately, the sort field request could just be contained in the initial query tag, perhaps like so:

pa:PageNum:ItemsPerPage vs pn:PageNum:ItemsPerPage
or
P:PageNum:ItemsPerPage vs p:PageNum:ItemsPerPage
or
a:PageNum:ItemsPerPage vs n:PageNum:ItemsPerPage (assuming that would not conflict with the existing use for a: as artist sort for direct search.

Regardless, this seems to be trivial in the current architecture as I understand it, and the alphanumeric tabbed entry could wait until it is trivial in the new design. If you were to code this up, I would then write up the corresponding web code and the UI, and post to Code. If I am correct, this is mostly just some simple math, a little code, and just using the existing architecture to serve pages as usual.
This would solve what from my experience is the number one stumbling block certain people have with server, both in general (sophisticated and unsophisticated users alike, who don't know what they want to sing, and thus have no search terms) as well as the specific (computer-phobes who freak out just at the sight of the computers) by allowing me to craft a new paradigm that more closely matches traditional books.
Hopefully just concentrating on this initial, easier implementation (that abandons alphanumeric-based entry for now) can bump this higher on the list, as this is the one key impediment to people using the server that I have found.
Last edited by DeusExMachina on Sun Mar 08, 2015 2:00 pm, edited 2 times in total.

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

Oh look, 100% of respondents desperately want this!!!
:)

jtara
Posts: 3
Joined: Sun Mar 08, 2015 12:25 pm

Re: server uploading specific pages to clients

Post by jtara »

It's always useful to be able to request data in pages! It's a common MVC meme, and I see the website, as quaint and ancient as it is, is really a SPA ("single page app") kind of thing which has become quite popular lately. It's consuming XML - didn't pay attention to whether it uses some JS library or XSLT to get to HTML.

It would be a nice addition to a RESTful API! We don't have one, though, so it gets a little awkward to implement. And I think Dave suggested that right now might not be the time for a new, RESTful API...

The web UI certainly could use some updating. Brushed panels are so.... well, I can't even remember! (It's not really the brushed panel, but that it has a definite opinion about desired screen size.) The good new there is, the way it works, nothing in the server really needs to be changed. (OK, except for pagination support!) Somebody could write a modern HTML page and modern Javascript with modern responsive CSS, and I assume it is trivial to drop those pages into the server. And it could consume the existing XML data.

So, yea, I'll vote for this - it is a good idea, if the web client is to be updated, it would be good to have pagination support. I might suggest, though, waiting until it can be incorporated into a new, more standardized API. I'm thinking myself that might be easier to implement in Python, not C. But I won't be the one doing it, so whatever works. I'd guess should at least wait for new SQL database.
----

As far as Sargent goes... meh - because I intend to do this kind of stuff onboard. I have to support live server search, but it isn't what I expect users to use most of the time. For live search, I am happy with the (default) 200-song limit.

But I expect most users will opt to store the entire Library onboard, where it can be searched more directly - and off-site to boot. I do plan on at least an A-Z, possibly (but probably not at first) infinite scroll, and the same kind of word search that kJams does on the server. So, I will have to implement the same concept, because I will need to get at parts of a result set.

FWIW, I am very sensitive to the "browsing" issue. There are "searchers". And there are "browsers". I'm a "browser". (Dave seemed surprised that there are browsers!) So, Sargent will be good for browsers. This change would make the server more browser-friendly.

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

Yeah, part of the motivation here is to allow the API to be left alone, FTMP, in a way which provides flexibility without necessitating core architecture changes. Leveraging the already existing search command and protocol architecture seems to provide that.
A similar rationale went into not really messing with the Web UI too much (that and my complete lack of artistic or design skills!). I have modified things somewhat, and my code is much less demanding regrading screen size, BUT, the one thing that is really impeding full size independence is the column headings in the main window, which are hard coded, and I have not had the time or skill set to really delve into getting them under control. But yes, your analysis of my motivation here is correct: to allow the basic UI to be not just malleable, but extensible, using the current server architecture and XML data stream.
Also, that the underlying db is going to change, as well as the API, is part of the rationale for doing it in this way, as it is straight forward with the current architecture (just add new data qualifier (currently there is "artist:", "name:", "a:", "n:") and tack on an extra piece of data on the end (the page number returned) that mimics just another expected returned record, and the server-side code should be minimal, since it is re-utilizing code that is already in place for search. At that point, switching to a new API (along with a new db architecture, will be trivial on the server end (but a massive pain in the butt on the client end!).
The other motivation, as discussed with Dave in other channels, is an attempt to reduce or eliminate the cognitive dissonance and usability challenges faced by some people confronted with the current UX. For some it is natural, but for others it is a HUGE impediment, to the point that some just walk away. Having made a detailed study of this, I have a totally different UX in mind, that removes this impediment for these people, as well as adding additional functionality to those comfortable with the current UX, but in order to do it, pagination support is a prerequisite.
These would be beneficial if implemented in a mobile client as well. I can discuss that separately if you'd be interested, or just forward the conversations.
As for Sargent (of which I know nothing) I'd have to say that if live server search only operated with a 200 song limit, that would pretty much preclude use by 99.9999% of all users, because kJams Pro and 2 are used in professional systems, with far more songs inter libraries, even if they are just streaming from the online stores, and kJams Lite doesn't support server, so no use for home users. (That said, I just realized you might mean the limit to number of songs returned. I have mine set to 1500 songs, and there has been no problem. So I am not sure how that is an issue here, since the time to transmit and format the search results was minimal relative to the search time.) Storing the db locally might work for a time, but you may very quickly run into reconciliation issues when the source db is modified, either because some metadata on an existing track has been changed (host discovers a misspelling, or is attempting to approach perfect metadata) songs are added, or songs are removed (corrupted .zip files, missing tracks, damaged CDGs etc.). You can of course use a simple checksum to spot a change (though you will have to get a server architecture change to even get that checksum data) but in order to track down the exact records involved might be lengthy and data intensive. Similar issues arise if you just force new users to download the db fresh each time or even just when checksums don't match (or kJams sets a flag telling you that the db has been modified) since every user in the venue will be doing so, often at the same time.
All this will be exacerbated once fast search is working, since the SQL database was roughly 5 times larger than the current file size.
That said, that is partly what I did in my new web client code that allows the user to specify "artist" or "song name" search via the UI, and then sort on columns. The sort takes place locally (and on a much smaller dataset) and is thus effectively instantaneous, as opposed to relying on the server, which, um, isn't.

As for the browsing issue, yes, it is a big one, especially for some people. Large enough for some that, as I mentioned, they just give up. I suspect there are others who see the set up, and don't even bother to come over!
That is the prime motivation for this feature, to allow a slight UX modification that can ease these people into the system, as well as providing the ability to browse, even for those with out those issues who have not yet been inspired or found their (meta)muse that evening.

dave
Site Admin
Posts: 6684
Joined: Sun Sep 18, 2005 8:02 am
Location: Seattle
Contact:

Re: server uploading specific pages to clients

Post by dave »

*sigh* it's on the list

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

Dave:
Did you get a chance to go over my latest post? Pretty sure the first part is only a little more time-intensive than the recent addition of the ability to add "artist:" search over server. And possibly the second part (alpha indexing) if my recent architecture suggestion (saving page indexes on initial library sort) is correct.
I can't emphasize enough how major a deal this would be for server, to REALLY set it apart from competing, very much more poorly implemented, offerings from other hosting apps.

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

Also, could we move this poll to a separate post? Not sure how many people are actually clicking through here, and of those few that do, even understand what is at issue.
Maybe something like:
"Adding ability to browse library sorted by artist or song name over server".
Last edited by DeusExMachina on Mon Mar 09, 2015 3:44 pm, edited 1 time in total.

dave
Site Admin
Posts: 6684
Joined: Sun Sep 18, 2005 8:02 am
Location: Seattle
Contact:

Re: server uploading specific pages to clients

Post by dave »

fine, i'm convinced. it's on the list. i'm a bit overwhelmed right now, i see "wall of text" and i freeze. when i come up for air and am ready to implement it, i'll come back here and read that short novel.

DeusExMachina
Posts: 1293
Joined: Sun Apr 20, 2008 9:57 am
Location: Pittsburgh, PA
Contact:

Re: server uploading specific pages to clients

Post by DeusExMachina »

:)
Never fear, the WOT (wall of text) is really pretty straight forward. TL:DR, it posits what might be a trivial implementation that will work both now and in the future db, either to just get scrub bar browsing, but also a simple hack to get alpha indexing to work!

dave
Site Admin
Posts: 6684
Joined: Sun Sep 18, 2005 8:02 am
Location: Seattle
Contact:

Re: server uploading specific pages to clients

Post by dave »

:oops:

karaokeniagara
Posts: 15
Joined: Thu Oct 15, 2015 11:32 am

Re: server uploading specific pages to clients

Post by karaokeniagara »

dave wrote:*sigh* it's on the list
I just hit the like button. Wink.

dave
Site Admin
Posts: 6684
Joined: Sun Sep 18, 2005 8:02 am
Location: Seattle
Contact:

Re: server uploading specific pages to clients

Post by dave »

the new wtkJams may suddenly make this idea obsolete. crossing fingers.

karaokeniagara
Posts: 15
Joined: Thu Oct 15, 2015 11:32 am

Re: server uploading specific pages to clients

Post by karaokeniagara »

dave wrote:the new wtkJams may suddenly make this idea obsolete. crossing fingers.
awesome. I currently use wtkjams ke at my shows and the singers love it.

Post Reply