Re: cachemgr output normalization

>> If the installation is connected to the Internet, we don't even need
>> to bundle the .js files, all that would be needed is a 1Kb-ish html
>> launcher.
>
> Where do you propose newly opened browsers will get the JS from to bootstrap
> the display?

Any hosting site / CDN willing to carry the js.

>>>> If this is difficult, for any reason, just put them somewhere on
>>>> squid-cache.org and hard-code the URLs in your Javascript, for now.
>>>>
>>>
>>> At first glance I think serving the JS page loader and a stub HTML page
>>> to
>>> run it as the http://*/squid-internal-manager/ 'index' URL is probably
>>> the
>>> user-friendliest way to roll it out.
>>>
>>>
>>> I think I spoke to you a while ago about implementing the /menu action
>>> with
>>> TXT + HTML format. But got stuck at the problem of generating right TXT
>>> in
>>> the workers or the HTTP headers output. And report aggregating
>>> duplication
>>> of the page.
>>
>> I'm aiming for something simpler now: rework the text output so that
>> it is still human-readable but a bit more machine-parser-friendly so
>> that transcoding (mostly transcoding tables) is simpler.
>>
>> The way I'd think to do the separation between data collection and
>> rendering is not really to use ASN.1 but structs. If multiple types
>> need to be supported it'll just have to be unions with a type
>> identifier attached. But in order to get there we need to reorganize
>> the actions' contents: most actions return two or more tables, this
>> needs to be reduced to one table per action, or it won't be renderable
>> by snmp.
>>
>>>>> 2- does cachemgr get engaged only via GET method or can we have it
>>>>> also answer to POST requests? The reason is that GET requests in
>>>>> javascript are subject to a same-origin policy, while POST are not. It
>>>>> would allow for multi-server monitoring and it would make point 1 a
>>>>> nice-to-have and not a requirement
>>>>
>>>> The difference is minor from Squid code point of view so we can support
>>>> both, eventually. IIRC, we do that in Co-Advisor, with little code (that
>>>> we would be happy to copy to Squid).
>>>
>>> I looked at this when advising Arthur on the shell backend handling. He
>>> needs POST as well.
>>>
>>> Cachemgr takes any method, but filtered GET for all the actions so far
>>> implemente. I think we can easily add any method we want for new actions.
>>> Or
>>> adjust the existing actions to handle non-GET.
>>
>> The actions don't really need to know for now: we can safely ignore
>> body-supplied parameters. If we don't, the adaptation should not be
>> done in the actions, that's too much duplication.
>>
>>>>> 3- we need to make the output from cachemgr handlers follow some
>>>>> common guidelines.
>>>>
>>>> Sure.
>>>>
>>>> How do you want to post-process that output in Javascript? Some
>>>> find-and-replace commands using regular expressions? Is it very
>>>> difficult to have action-specific post-processing?
>>
>> Find-and-replace, inspired to cachemgr.cgi.
>>
>>> The JS should make use of the format parameter to do format-specific
>>> processing. Like CGI does action-specific processing right now.
>>
>> Current Js is limited to browser-based rendering. Think of it as an
>> alternative to cachemgr.cgi, not as a self-contained tool.
>
> I was imagining that you had the JS doing AJAX / XHR in the background to
> load the reports yes?

Yes.

>  So trying to load the report in format A. ON success passing it to the
> format-A display function. On fail trying with format B...
>
> Instead of deciding up front based on the action name.
>
> That way the JS would be truly independent of how far we had managed to
> convert the reports. Like cachemgr is not.

It currently uses the same logic as cachemgr.cgi in a simplifiedform.
something like: "if there's a tab, it's a table".

>>> The existing TXT format can be dumped wholesale straight into PRE tags
>>> and
>>> the actions updated to more useful formats one by one. This resolves the
>>> back-compat and third-party script problems as text can be phased
>>> out/upgraded over the long term.
>>>
>>>  * For tabular data CSV, TSV, HTML, XML formats are all commonly
>>> supported
>>> and useful under different scenarios.
>>>  * For list data XML or HTML is probably best.
>>>  * For key=value data XML, HTML, or TXT is probably best.
>>
>> That's the point: key = value data is really a two-columns table.
>> Which is rendered as key = value probably due to historic reasons.
>> That's exactly what I'd like to change.
>
> The k-v entries I've had the change to look at were actually a.b.c.key=value
> which is more tree structured than K-V pair.

"a.b.c.key" = "key with a hierarchic hint" IMVHO

>>>>> This poses a problem of
>>>>> compatibility with third-party software. We can either have a
>>>>> transition phase where we duplicate actions, or we can just decide
>>>>> that we don't have the resources to care, and we just warn the authors
>>>>> that we know of about our intentions so that they have time to adapt.
>>>>
>>>> Indeed. Perhaps this should be discussed on squid-users as well.
>>>>
>>>
>>> This is where I greatly favour the format upgrade. We can both warn them
>>> the
>>> TXT is going to become human-only view, and that there is more efficient
>>> machine-readable formats available for their immediate use.
>>
>> That's for phase-two. Phase-one would be to make txt more
>> machine-friendly.
>
> Altering TXT without some alternative and lead time already being available
> for people to convert to could cause problems.
>  I know it seems a bit messy now with each txt report having its own
> standard of syntax. But the people depending on it dont really care much,
> they already have code setup to handle those syntax.
>
> PS. sorry if I confused you with talks earlier and/or with Alex, about being
> free to update the reports. I meant the values and fields reported not the
> deeper layouts or syntax of the report itself.
>  If you get what I mean?

I don't think I do. Maybe we need to talk about this on IRC and over a
live demo (currently done on eu, except for some css fixups).
It is password-protected, those interested please raise a hand so that
I can grant them access.

>>>>> This is also in my opinion a prerequisite step to support multiple
>>>>> output formats in cachemgr.
>>>>
>>>> Is there consensus that Squid itself should support multiple output
>>>> formats? I kind of doubt it is the right thing to do in general. If
>>>> Squid outputs easy-to-parse, consistent data, other applications can
>>>> post-process and beautify it in many different ways.
>>>>
>>>
>>>  From me yes.
>>>
>>> I also see and agree that opening the door to anything at all is a bad
>>> idea.
>>> We can and should provide a small restricted set of suitable formats for
>>> the
>>> data based on its type table/list etc. Such as the ones I mention up
>>> above.
>>
>> I'd add JSON, but apart from that yes.
>
> Yeah. I thought about that after sending.
>
>>
>>>>> I'm willing to spend the time to do this if we agree that it should be
>>>>> done.
>>>>
>>>> Yes, the output should be standardized.
>>>>
>>>
>>> Agreed.
>>>
>>> The first step though, is standardizing and checking that all actions are
>>> upgraded to use a single internal-only format for data transfer from
>>> workers
>>> to somewhere for assembly/aggregation and transformation the other
>>> formats
>>> as needed. ASN was proposed earlier. Yes?
>>
>> That's second step :)
>
> I disagree. You would then be:
>  1 altering the worker report format to X
>  2 altering the worker report format to internal syntax
>  3 replicating step (1) in the coordinator
>
> Better to save work+time and just do the second two steps as one auditable
> unit (per action if they get big).
>
>>
>> First step, standardize textual representation in output with current
>> infrastructure
>> Second step, restructure actions (one table per action)
>
> Adding new actions with structured grouping/path syntax:
>  .../menu
>  .../menu/peers
>  .../menu/kids
>  .../cache/fqdn
>  .../cache/ip
>  .../cache/usernames
>  ...
>
>> Third step, separate internal and extenral representation
>> Fourth step, add more formats.

these are just labels :) The separator needn't be slash, even.

-- 
    /kinkie